More Web Proxy on the site http://driver.im/

article

Design of experiments in computer performance evaluation

Author:

Martin SchatroffAuthors Info & Claims

IBM Journal of Research and Development, Volume 25, Issue 6

Pages 848 - 859

https://doi.org/10.1147/rd.256.0848

Published: 01 November 1981 Publication History

Abstract

Techniques of statistical design of experiments have been successfully employed for many decades in a variety of applications in industry, agriculture, medicine, psychology, and other physical and social sciences. Their aim is to provide scientific and eficient means of studying the effects, on one or more variables of interest, of varying multiple controllable factors in an experiment. These techniques have not been widely used in the study of computer systems, although they can potentially have as large an impact as they have had in other fields. The purpose of this paper is to review some of the basic concepts underlying the statistical design and analysis of experiments and to illustrate them by means of examples drawn from studies of computer system performance. The examples include comparisons of alternate page replacement and free storage management algorithms, optimization of a scheduler, and validation of a system simulation model.

References

[1]

W. G. Cochran and G. M. Cox, Experimental Designs (Second Edition), John Wiley & Sons, Inc., New York, 1957.

[2]

The Design and Analysis of Industrial Experiments, O. L. Davies, Ed., Oliver and Boyd, London, 1954.

[3]

G. E. P. Box and J. S. Hunter, "The 2^k-p Fractional Factorial Designs, Part 1," Technometrics 3, 311-351 (1961).

[4]

D. J. Finney, "The Fractional Replication of Factorial Arrangements," Ann. Eugenics 12, 291-301 (1945).

[5]

R. L. Plackett and J. P. Burman, "The Design of Optimum Multifactorial Experiments," Biometrika 33, 328 (1946).

[6]

S. Addelman and O. Kempthorne, "Orthogonal Main-effect Plans and Orthogonal Arrays of Rank 2," Ann. Math. Statist. 32, 1167-1176 (1961).

[7]

S. Addelman, "Orthogonal Main-Effect Plans for Asymmetrical Factorial Experiments," Technometrics 4, 21-46 (1962).

[8]

B. H. Margolin, "Non-orthogonal Main-effect Designs for Asymmetrical Factorial Experiments," Roy. Statist. Soc. B 34, 431 (1972).

[9]

Jack P. C. Kleijnen, "Screening Designs for Poly-factor Experimentation," Technometrics 17, 487 (1975).

[10]

B. H. Margolin, R. I. Parmelee, and M. Schatzoff, "Analysis of Free-Storage Algorithms," IBM Syst. J. 10, 283-304 (1971).

Digital Library

[11]

Y. Bard, "Experimental Evaluation of System Performance," IBM Syst. J. 12, 302-314 (1973).

Digital Library

[12]

T. W. Anderson, The Statistical Analysis of Time Series, John Wiley & Sons, Inc., New York, 1971.

[13]

M. Schatzoff and C. C. Tillman, "Design of Experiments in Simulator Validation," IBM J. Res. Develop. 19, 252-262 (1975).

Digital Library

[14]

G. E. P. Box and D. B. Wilson, "On the Experimental Attainment of Optimum Conditions," Roy. Statist. Soc. B 3, 1-45 (1951).

[15]

M. Schatzoff and P. G. Bryant, "Regression Methods in Perfomance Evaluation: Some Comments on the State of the Art," Proceedings of Computer Science and Statistics, 7th Annual Symposium on the Interface, William J. Kennedy, Ed., Iowa State University, Ames, IA, 48-57, October 18 and 19, 1973.

[16]

Y. Bard, "An Experimental Approach to System Tuning," Proceedings of the International Symposium on Computer Performance Modeling, Measurement and Evaluation, Peter P. S. Chen and Mark Franklin, Eds., Harvard University, Cambridge, MA, March 29-31, 1976.

[17]

Y. Bard, "Application of the Page Survival Index (PSI) to Virtual-memory System Performance," IBM J. Res. Develop. 19, 212-220 (1975).

Digital Library

Cited By

Pieterse VCleophas LMasinde M(2017)Benchmarking optimised algorithms for transitive closureProceedings of the South African Institute of Computer Scientists and Information Technologists10.1145/3129416.3129425(1-10)Online publication date: 26-Sep-2017
https://dl.acm.org/doi/10.1145/3129416.3129425
Panti MValenti S(1992)A modulus oriented hash function for the construction of minimal perfect tablesACM SIGPLAN Notices10.1145/141018.14103527:11(33-38)Online publication date: 1-Nov-1992
https://dl.acm.org/doi/10.1145/141018.141035
Berry R(1992)Computer Benchmark Evaluation and Design of ExperimentsIEEE Transactions on Computers10.1109/12.16660541:10(1279-1289)Online publication date: 1-Oct-1992
https://dl.acm.org/doi/10.1109/12.166605
Show More Cited By

Recommendations

Computer Benchmark Evaluation and Design of Experiments: A Case Study

The author defines installation tuning and promotes it as an important area of concern for computer configurations. Examples include deciding on the paging configuration for a particular workload and partitioning available memory into system and user ...
Statistical methods for computer performance evaluation

Computer performance has traditionally played a fundamental role in computer system design. Performance measurements and their quantitative analyses are useful in understanding and enhancing a computer system and in comparing different systems. Owing to ...
Designs for computer experiments

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IBM Journal of Research and Development

IBM Journal of Research and Development Volume 25, Issue 6

November 1981

124 pages

ISSN:0018-8646

Issue’s Table of Contents

Publisher

IBM Corp.

United States

Publication History

Published: 01 November 1981

Revised: 01 June 1981

Received: 21 July 1980

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
1
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pieterse VCleophas LMasinde M(2017)Benchmarking optimised algorithms for transitive closureProceedings of the South African Institute of Computer Scientists and Information Technologists10.1145/3129416.3129425(1-10)Online publication date: 26-Sep-2017
https://dl.acm.org/doi/10.1145/3129416.3129425
Panti MValenti S(1992)A modulus oriented hash function for the construction of minimal perfect tablesACM SIGPLAN Notices10.1145/141018.14103527:11(33-38)Online publication date: 1-Nov-1992
https://dl.acm.org/doi/10.1145/141018.141035
Berry R(1992)Computer Benchmark Evaluation and Design of ExperimentsIEEE Transactions on Computers10.1109/12.16660541:10(1279-1289)Online publication date: 1-Oct-1992
https://dl.acm.org/doi/10.1109/12.166605
Kleijnen J(1988)Analyzing Simulation Experiments with Common Random NumbersManagement Science10.5555/3214455.321445934:1(65-74)Online publication date: 1-Jan-1988
https://dl.acm.org/doi/10.5555/3214455.3214459

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents