More Web Proxy on the site http://driver.im/

keynote

Top500 versus sustained performance: the top problems with the top500 list - and what to do about them

Author:

William T.C. KramerAuthors Info & Claims

PACT '12: Proceedings of the 21st international conference on Parallel architectures and compilation techniques

Pages 223 - 230

https://doi.org/10.1145/2370816.2370850

Published: 19 September 2012 Publication History

Abstract

A popular U.S. talk show host uses "top 10" lists to critique events and culture every night. Our HPC industry is captivated by another list, the TOP500 list, as a way to track HPC systems' performance based on FLOPS/S assessed by a single, long-lived benchmark-Linpack. The TOP500 list has grown in influence because of its value as a marketing tool. It simplistically, but unrealistically, describes performance of HPC systems. The proponents have advocated for the TOP500 list for different reasons at different times. This paper critiques the Top 10 problems with the TOP500 list and provides suggestions on how to correct those shortcomings. It discusses why the TOP500 list is limiting the impact of HPC systems on real problems and other metrics that may be more meaningful and useful to represent the real effectiveness and value of HPC systems.

References

[1]

Dongarra, Jack. Performance of Various Computers Using Standard Linear Equations Software. Computer Science, University of Tennessee, Knoxville TN, 37996: University of Tennessee, 1985.

[2]

Top 500 List. 1993-2012. http://www.top500.org (accessed 2012.

[3]

http://www.netlib.org/Linpack.

[4]

Linpack Download. 2008. http://www.netlib.org/LINPACK (accessed 2008).

[5]

Kramer, William T. C., PERCU Results in a Reawakened Relationship for NERSC and Cray, Cray User Group (CUG 2007) Conference, Seattle, WA, May 2007.

[6]

Kramer, William. Blue Waters - A Super System for Super Challenges. Proceedings of the Cray User Group, Stuttgart, Germany, April 29-May 3, 2012.

[7]

NERSC SSP Project. NERSC Projects. http://www.nersc.gov/projects/ssp.php (accessed 2011).

[8]

Kramer, William. PERCU: A Holistic Method for Evaluating High Performance Computing Systems, A Dissertation, University of California Berkeley, October 2008. http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-143.pdf.

Digital Library

[9]

Carrington, Laura, M. Laurenzano, Allan Snavely, Roy Campbell, and Larry Davis. How Well Can Simple Metrics Represent the Performance of HPC Applications? SC 05 - The High Performance Computing, Storage, Networking and Analysis Conference 2005. Seattle, WA: Association of Computing Machinary (ACM), 2005.

Digital Library

[10]

Tikir, M., L. Carrington, E. Strohmaier, and A. Snavely. A Genetic Algorithms Approach to Modeling the Performance of Memory-bound Computations. Proceedings of SC07. Reno, NV: Association of Computing Machinery (ACM), 2007.

Digital Library

[11]

SPEC Benchmarks. 2000. http://www.spec.org (accessed 2008).

[12]

Bailey, David H., and et. al. The NAS Parallel Benchmarks. Intl. Journal of Supercomputer Applications vol 5, no. 3 (Fall 1991): 66--73.

Digital Library

[13]

Simon, Horst, and Erich Strohmaier. Statistical Analysis of NAS Parallel Benchmarks and LINPACK Results. Vol. 919, in Lecture Notes In Computer Science, edited by Bob Hertzberger and Guiseppe Serazzi, 626--633. London: Springer-Verlag, 1995.

Digital Library

[14]

High Performance Technology Insertion 2006 (TI-06). DOD Modernization Program. 2005. http://www.fbodaily.com/archive/2005/05-May/08-May-2005/FBO-00802613.htm (accessed 2005).

[15]

Culler, David E., and Jaswinder Pal Singh. Parallel Computer Architecture: A Hardware/Software Approach. First. Edited by Denise P. M. Penrose. San Francisco, CA: Morgan Kaufmann Publishers, Inc., 1999.

Digital Library

[16]

Dongarra, Jack, Reduced Linpack to Keep the Run Time Manageable for Future TOP500 Lists, invited talk, International Supercomputing Conference, ISC 12, Homburg, Germany, June 17-21, 2012.

[17]

The Networking and Information Technologies Research and Development Program, Supplement to the President's Budget FY1993 to FY 2013, http://www.nitrd.gov/pubs/2013supplement/FY13NITRDSupplement.pdf.

[18]

Consumer Price Index - Bureau of Labor Statistics. www.bls.gov/cpi/, (Accessed April 2012).

[19]

Moore, Gordon E., Cramming more components onto integrated circuits. Electronics Magazine, 1965.

[20]

Saini, Subach and David Bailey, NAS Parallel Benchmark Results (Version 1.0) 11-96. NAS Technical Report 96-18, Nasa Ames, tables 13--17. http://www.nas.nasa.gov/assets/pdf/techreports/1996/nas-96-018.pdf - other data is available in similar reports.

[21]

Smith, J. E. Characterizing Computer Performance with a Single Number. Communications of the ACM (Association of Computing Machinery) 31, no. 10 (October 1988): 1202--1206.

Digital Library

[22]

Lilja, David. Measuring Computer Performance: A Practitioner's Guide. Cambridge University Press, 2000.

Digital Library

[23]

Dongarra, Jack. Performance of Various Computers Using Standard Linear Algebra Software in a Fortran Environment. HPCC. http://www.netlib.org/benchmarks/performance.ps (accessed 2012).

[24]

McMahon, F. The Livermore Fortran Kernels: A computer test of numerical performance range. Technical Report, Lawrence Livermore National Laboratory, Livermore, CA: University of California, 1986.

[25]

Mashey, John R. War of the Benchmark Means: Time for a Truce. ACM SIGARCH Computer Architecture News (Association for Computing Machinery) 32, no. 4 (September 2004).

Digital Library

[26]

Bucher, Ingrid, and Joanne Martin. Methodology for Characterizing a Scientific Workload. Technical Report, Los Alamos, NM 87545: Los Alamos National Laboratory, 1982.

[27]

Gregory H. Bauer, Torsten Hoefler, William T. Kramer and Robert A. Fiedler. Analyses and Modeling of Applications Used to Demonstrate Sustained Petascale Performance on Blue Waters, Proceedings of the Cray User Group, Stuttgart, Germany, April 29 - May 3, 2012.

Cited By

Voss NKwaadgras BMencer OLuk WGaydadjiev G(2021)On Predictable Reconfigurable System DesignACM Transactions on Architecture and Code Optimization10.1145/343699518:2(1-28)Online publication date: 9-Feb-2021
https://dl.acm.org/doi/10.1145/3436995
Savin GShabanov BTelegin PBaranov A(2019)Joint Supercomputer Center of the Russian Academy of Sciences: Present and FutureLobachevskii Journal of Mathematics10.1134/S199508021911027140:11(1853-1862)Online publication date: 27-Nov-2019
https://doi.org/10.1134/S1995080219110271
Bauer GBode BEnos JKramer WLathrop SMendes CSisneros R(2018)Best practices and lessons from deploying and operating a sustained-petascale systemProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291727(1-12)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291727
Show More Cited By

Index Terms

Top500 versus sustained performance: the top problems with the top500 list - and what to do about them
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
2. General and reference
  1. Cross-computing tools and techniques

Recommendations

Second Generation Quad-Core Intel Xeon Processors Bring 45 nm Technology and a New Level of Performance to HPC Applications
ICCS '08: Proceedings of the 8th international conference on Computational Science, Part I

The second generation of Quad-Core Intel® Xeon® processors was launched on November 12th 2007. In this paper we take a look at what the new 45 nm based Quad-Core Intel Xeon Processor brings to high performance computing. We compare an Intel Xeon 5300 ...
Evaluating performance of new quad-core Intel®Xeon®5500 family processors for HPC
PPAM'09: Proceedings of the 8th international conference on Parallel processing and applied mathematics: Part I

In this paper we take a look at what the new Quad-Core Intel Xeon Processor code name Nehalem brings to high performance computing. We compare Intel Xeon 5400 series based system with a server utilizing his successor the new Intel Xeon X5560. We compare ...
Performance evolution and power benefits of cluster system utilizing quad-core and dual-core Intel Xeon processors
PPAM'07: Proceedings of the 7th international conference on Parallel processing and applied mathematics

Multi-core processors represent an evolutionary change in conventional computing as well setting the new trend for high performance computing. The chip-level multiprocessing architectures with a large number of cores continue to offer dramatically ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

PACT '12: Proceedings of the 21st international conference on Parallel architectures and compilation techniques

September 2012

512 pages

ISBN:9781450311823

DOI:10.1145/2370816

General Chairs:
Pen-Chung Yew
University of Minnesota
,
Sangyeun Cho
University of Pittsburgh
,
Program Chairs:
Luiz DeRose
Cray, Inc.
,
David J. Lilja
University of Minnesota

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

IFIP WG 10.3: IFIP WG 10.3
SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE CS TCPP: IEEE Computer Society Technical Committee on Parallel Processing
IEEE CS TCAA: IEEE CS technical committee on architectural acoustics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Keynote

Conference

PACT '12

Sponsor:

IFIP WG 10.3
SIGARCH
IEEE CS TCPP
IEEE CS TCAA

PACT '12: International Conference on Parallel Architectures and Compilation Techniques

September 19 - 23, 2012

Minnesota, Minneapolis, USA

Acceptance Rates

Overall Acceptance Rate 121 of 471 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
250
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Voss NKwaadgras BMencer OLuk WGaydadjiev G(2021)On Predictable Reconfigurable System DesignACM Transactions on Architecture and Code Optimization10.1145/343699518:2(1-28)Online publication date: 9-Feb-2021
https://dl.acm.org/doi/10.1145/3436995
Savin GShabanov BTelegin PBaranov A(2019)Joint Supercomputer Center of the Russian Academy of Sciences: Present and FutureLobachevskii Journal of Mathematics10.1134/S199508021911027140:11(1853-1862)Online publication date: 27-Nov-2019
https://doi.org/10.1134/S1995080219110271
Bauer GBode BEnos JKramer WLathrop SMendes CSisneros R(2018)Best practices and lessons from deploying and operating a sustained-petascale systemProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291727(1-12)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291727
Bauer GBode BEnos JKramer WLathrop SMendes CSisneros R(2018)Best practices and lessons from deploying and operating a sustained-petascale systemProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC.2018.00056(1-12)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.1109/SC.2018.00056
Radulovic MAsifuzzaman KCarpenter PRadojković PAyguadé E(2018)HPC Benchmarking: Scaling Right and Looking Beyond the AverageEuro-Par 2018: Parallel Processing10.1007/978-3-319-96983-1_10(135-146)Online publication date: 1-Aug-2018
https://doi.org/10.1007/978-3-319-96983-1_10
Kwack JArnold GMendes CBauer G(2018)Roofline analysis with Cray performance analysis tools (CrayPat) and roofline‐based performance projections for a future architectureConcurrency and Computation: Practice and Experience10.1002/cpe.496331:16Online publication date: 25-Sep-2018
https://doi.org/10.1002/cpe.4963
Kwack JBauer G(2017)HPCG and HPGMG benchmark tests on multiple program, multiple data (MPMD) mode on Blue Waters—A Cray XE6/XK7 hybrid systemConcurrency and Computation: Practice and Experience10.1002/cpe.429830:1Online publication date: 9-Oct-2017
https://doi.org/10.1002/cpe.4298
Selinger ARupp KSelberherr SWeinbub JBaboulin MThacker WPolok L(2016)Evaluation of mobile ARM-based SoCs for high performance computingProceedings of the 24th High Performance Computing Symposium10.22360/SpringSim.2016.HPC.022(1-7)Online publication date: 3-Apr-2016
https://dl.acm.org/doi/10.22360/SpringSim.2016.HPC.022
Xiong HZeng GWang WWu CWang Y(2016)Upper Limit Analysis of Scalable Parallel Computing on the Premise of Reliability RequirementIETE Technical Review10.1080/02564602.2015.112620333:6(573-583)Online publication date: Mar-2016
https://doi.org/10.1080/02564602.2015.1126203
Breuer AHeinecke ARettenberger SBader MGabriel APelties C(2014)Sustained Petascale Performance of Seismic Simulations with SeisSol on SuperMUCProceedings of the 29th International Conference on Supercomputing - Volume 848810.1007/978-3-319-07518-1_1(1-18)Online publication date: 22-Jun-2014
https://dl.acm.org/doi/10.1007/978-3-319-07518-1_1

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents