More Web Proxy on the site http://driver.im/

article

Benchmarking with Real Industrial Applications: The SPEC High-Performance Group

Authors:

Rudolf Eigenmann,

Siamak HassanzadehAuthors Info & Claims

IEEE Computational Science & Engineering, Volume 3, Issue 1

Pages 18 - 23

https://doi.org/10.1109/99.486757

Published: 01 March 1996 Publication History

Abstract

Simulations are only as good as the accuracy of the model. A truism, but one seemingly forgotten at times in the "benchmarking" of computer systems--the judging of their speed and performance running standardized programs. For really useful comparative benchmarking, not only must the conditions of the experiment be controlled, but the premises of the experiment must be valid. That is, the standard benchmark programs must be a reasonably good approximation of applications that real users might want to run on real machines under real working conditions. Too much simplification in the codes, and the results are of limited use in predicting performance.SPEC, the Standard Performance Evaluation Corporation, has been a leader in evaluating the performance of workstations since its formation in the late 1980s. Now a group within SPEC is turning its attention to real high-performance computing, and it's emphasizing the use of realistic scientific and industrial codes as benchmarks. A seismic code for oil exploration and a computational chemistry program are the first two components of the SPEChpc96 suite. SPEC hopes that interest and participation from users will help speed up the somewhat sluggish growth of real HPC performance.

References

[1]

G. Cybenko, et al., "Supercomputer Performance Evaluation and the Perfect Benchmarks," Proc. ICS '90: Int'l Conf. on Supercomputing, ACM Press, New York, 1990, pp. 254-266.

Digital Library

[2]

M. Berry, et al., "The Perfect Club Benchmarks: Effective Performance Evaluation of Supercomputers," Int'l J. Supercomputer Applications, Vol. 3, No. 3, Fall 1989, pp. 5-40.

Digital Library

[3]

L. Pointer, "Perfect: Performance Evaluation for Cost-Effective Transformations Report 2," Tech. Report 964, Center for Supercomputing Research and Development, Univ. of Illinois at Urbana-Champaign, 1990.

[4]

C.C. Mosher and S. Hassanzadeh, "ARCO Seismic Processing Performance Evaluation Suite, User's Guide," tech. report, Atlantic Richfield, Plano, Texas, 1993.

[5]

M. Ginsberg, "Creating an Automotive Industry Benchmark Suite for Assessing theEffectiveness of High-Performance Computers," Proc. Ninth Int'l Conf. Vehicle Structural Mechanics and CAE, Soc. of Automotive Engineers, Warrendale, Penn., 1995, pp. 381-390.

[6]

R.W. Hockney and M. Berry, eds., "Parkbench Report: Public International Benchmarking for Parallel Computers," Scientific Programming, Vol. 3, No. 2, 1994, pp. 101-146.

Digital Library

[7]

F. McMahon, "The Livermore Kernels: A Computer Test of the Numerical Performance Range," in Performance Evaluation of Supercomputers, J.L. Martin, ed., Elsevier Science, Amsterdam, 1988, pp. 143-186.

[8]

R.P. Weicker, "An Overview of Common Benchmarks," Computer, Vol. 23, No. 12, Dec. 1990, pp. 65-75.

Digital Library

[9]

J.J. Dongarra, "The Linpack Benchmark: An Explanation," in Evaluating Supercomputers, A.J. van der Steen, ed., Chapman and Hall, London, 1990, pp. 1-21.

Digital Library

[10]

A.J. van der Steen, "The Benchmark of the EuroBen Group," Parallel Computing, Vol. 17, 1991, pp. 1,211-1,221.

Digital Library

[11]

C.A. Addison, et. al., "The Genesis Distributed-Memory Benchmarks," in Computer Benchmarks, J.J. Dongarra and W. Gentzsch, eds., Elsevier Science, Amsterdam, 1991, pp. 257-271.

Digital Library

[12]

D.H. Bailey, et al., "NAS Parallel Benchmark Results," Proc. Supercomputing '92, IEEE Computer Soc. Press, Los Alamitos, Calif., 1992, pp. 386-393.

Digital Library

[13]

J.P. Singh W.-D. Weber and A. Gupta, "Splash: Stanford Parallel Applications for Shared-Memory," Computer Architecture News, Vol. 20, No. 1, 1992, pp. 5-44.

Digital Library

[14]

M.W. Schmidt, et al., "The General Atomic and Molecular Electronics Structure Systems," J. Computational Chemistry, Vol. 14, No. 11, 1993, pp. 1,347-1,363.

Digital Library

[15]

A. Beguelin, et al., "PVM: Experiences, Current Status and Future Direction," Proc. Supercomputing '93, IEEE Computer Soc. Press, Los Alamitos, Calif., 1993, p. 765-766.

[16]

B. Pottenger and R. Eigenmann, "Targeting a Shared-Address-Space Version of the Seismic Benchmark Seis1.1," Tech. Report 1456, Center for Supercomputing Research and Development,Univ. of Illinois at Urbana-Champaign, 1995.

Cited By

Afzal AHager GWellein G(2023)SPEChpc 2021 Benchmarks on Ice Lake and Sapphire Rapids Infiniband Clusters: A Performance and Energy Case StudyProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624197(1245-1254)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624197
Varghese BWang NBermbach DHong CLara EShi WStewart C(2021)A Survey on Edge Performance BenchmarkingACM Computing Surveys10.1145/344469254:3(1-33)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3444692
Zhang JFigueiredo R(2006)Application classification through monitoring and learning of resource consumption patternsProceedings of the 20th international conference on Parallel and distributed processing10.5555/1898953.1899074(144-144)Online publication date: 25-Apr-2006
https://dl.acm.org/doi/10.5555/1898953.1899074
Show More Cited By

Recommendations

Application-specific benchmarking
Renaissance: benchmarking suite for parallel applications on the JVM
PLDI 2019: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation

Established benchmark suites for the Java Virtual Machine (JVM), such as DaCapo, ScalaBench, and SPECjvm2008, lack workloads that take advantage of the parallel programming abstractions and concurrency primitives offered by the JVM and the Java Class ...
Benchmarking data warehouses

Database benchmarks can either help users in comparing the performances of different systems, or help engineers in testing the effect of various design choices. In the field of data warehouses, the Transaction Processing Performance Council's standard ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Computational Science & Engineering

IEEE Computational Science & Engineering Volume 3, Issue 1

March 1996

89 pages

ISSN:1070-9924

Issue’s Table of Contents

Copyright © Copyright © 1996 IEEE. All Rights Reserved.

Publisher

IEEE Computer Society Press

Washington, DC, United States

Publication History

Published: 01 March 1996

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Afzal AHager GWellein G(2023)SPEChpc 2021 Benchmarks on Ice Lake and Sapphire Rapids Infiniband Clusters: A Performance and Energy Case StudyProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624197(1245-1254)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624197
Varghese BWang NBermbach DHong CLara EShi WStewart C(2021)A Survey on Edge Performance BenchmarkingACM Computing Surveys10.1145/344469254:3(1-33)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3444692
Zhang JFigueiredo R(2006)Application classification through monitoring and learning of resource consumption patternsProceedings of the 20th international conference on Parallel and distributed processing10.5555/1898953.1899074(144-144)Online publication date: 25-Apr-2006
https://dl.acm.org/doi/10.5555/1898953.1899074
Eigenmann RGaertner GJones WSaito HWhitney B(2002)SPEC HPC2002Proceedings of the 4th International Symposium on High Performance Computing10.5555/646349.690720(7-10)Online publication date: 15-May-2002
https://dl.acm.org/doi/10.5555/646349.690720
Bova SBreshears CGabb HKuhn BMagro BEigenmann RGaertner GSalvini SScott H(2001)Parallel programming with message passing and directivesComputing in Science and Engineering10.1109/5992.9471053:5(22-37)Online publication date: 15-Sep-2001
https://dl.acm.org/doi/10.1109/5992.947105
Hey TLancaster D(2000)The Development of Parkbench and Performance PredictionInternational Journal of High Performance Computing Applications10.1177/10943420000140030414:3(205-215)Online publication date: 1-Aug-2000
https://dl.acm.org/doi/10.1177/109434200001400304
Eigenmann RHoeflinger JPadua D(1998)On the Automatic Parallelization of the Perfect Benchmarks®IEEE Transactions on Parallel and Distributed Systems10.1109/71.6552389:1(5-23)Online publication date: 1-Jan-1998
https://dl.acm.org/doi/10.1109/71.655238

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents