An experiment specification language for goal-driven, automated performance evaluations
Pages 1043 - 1048
Abstract
In performance engineering, experimentation is used to compare design alternatives, identify bottlenecks, or infer prediction functions. In many cases, performance analysts apply individual, unstructured approaches for the definition of experiments which hampers reusability, collaboration, and automation. In this paper, we introduce an experiment specification language that supports performance analysts in capturing all information required to implement a systematic performance evaluation process. The language design specifically aims at facilitating reusability, collaboration, and automation. The presented approach is applied in different scenarios at SAP development departments.
References
[1]
D. Abramson, J. Giddy, and L. Kotler. High performance parametric modeling with nimrod/g: Killer application for the global grid? In Proceedings of the 14th International Parallel & Distributed Processing Symposium (IPDPS'00), pages 520--528. IEEE Computer Society, 2000.
[2]
D. Abramson, R. Sosic, J. Giddy, and B. Hall. Nimrod: a tool for performing parametrised simulations using distributed workstations. In Proceedings of the 4th IEEE International Symposium on High Performance Distributed Computing, HPDC '95, pages 112--, Washington, DC, USA, 1995. IEEE Computer Society.
[3]
J. H. Friedman. Multivariate Adaptive Regression Splines. Annals of Statistics, 19(1):1--141, 1991.
[4]
Y. E. Ioannidis, M. Livny, S. Gupta, and N. Ponnekanti. Zoo: A desktop experiment management environment. In T. M. Vijayaraman, A. P. Buchmann, C. Mohan, and N. L. Sarda, editors, VLDB'96, Proceedings of 22th International Conference on Very Large Data Bases, September 3--6, 1996, Mumbai (Bombay), India, pages 274--285. Morgan Kaufmann, 1996.
[5]
R. Jain. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley professional computing. Wiley, 1991.
[6]
K. L. Karavanic and B. P. Miller. Experiment management support for performance tuning. In Proceedings of the 1997 ACM/IEEE conference on Supercomputing (CDROM), Supercomputing '97, pages 1--10, New York, NY, USA, 1997. ACM.
[7]
H. Koziolek, B. Schlich, C. Bilich, R. Weiss, S. Becker, K. Krogmann, M. Trifu, R. Mirandola, and A. Koziolek. An industrial case study on quality impact prediction for evolving service-oriented software. In Proceedings of the 33rd International Conference on Software Engineering, ICSE '11, pages 776--785, New York, NY, USA, 2011. ACM.
[8]
B. P. Miller, M. D. Callaghan, J. M. Cargille, J. K. Hollingsworth, R. Bruce, I. Karen, L. Karavanic, K. Kunchithapadam, and T. Newhall. The paradyn parallel performance measurement tools. IEEE Computer, 1995.
[9]
M. Natrella. NIST/SEMATECH e-Handbook of Statistical Methods. NIST/SEMATECH, 2012.
[10]
R. Prodan and T. Fahringer. Zenturio: a grid middleware-based tool for experiment management of parallel and distributed applications. Journal of Parallel and Distributed Computing, 64(6):693--707, June 2004.
[11]
R. Prodan and T. Fahringer. Zen: a directive based experiment specification language for performance and parameter studies of parallel scientific applications. International Journal of High Performance Computing and Networking, 3(2/3):103--121, Nov. 2005.
[12]
T. Schneider. SAP Performance Optimization Guide: Analyzing and Tuning SAP Systems. Galileo Pr Inc., 2006.
[13]
C. U. Smith and L. G. Williams. Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Addison-Wesley Professional, 1st edition, 2001.
[14]
D. Westermann, J. Happe, M. Hauck, and C. Heupel. The performance cockpit approach: A framework for systematic performance evaluations. In Proceedings of the 36th EUROMICRO SEAA 2010. IEEE CS, 2010.
[15]
D. Westermann, J. Happe, R. Krebs, and R. Farahbod. Automated inference of goal-oriented performance prediction functions. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ASE 2012, pages 190--199, New York, NY, USA, 2012. ACM.
[16]
J. Worringen. Experiment management and analysis with perfbase. In IEEE International Conference on Cluster Computing (CLUSTER 2005), pages 1--11. IEEE, 2005.
Index Terms
- An experiment specification language for goal-driven, automated performance evaluations
Recommendations
A specification language for organisational performance indicators
A specification language for performance indicators and their relations and requirements is presented and illustrated for a case study in logistics. The language can be used in different forms, varying from informal, semiformal, graphical to formal. A ...
Rule-based automatic software performance diagnosis and improvement
There are many advantages to analyzing performance at the design level, rather than waiting until system testing. However the necessary expertise in making and interpreting performance models may not be available, and the time for the analysis may be ...
Efficient experiment selection in automated software performance evaluations
EPEW'11: Proceedings of the 8th European conference on Computer Performance EngineeringThe performance of today's enterprise applications is influenced by a variety of parameters across different layers. Thus, evaluating the performance of such systems is a time and resource consuming process. The amount of possible parameter combinations ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2013
2124 pages
Copyright © 2013 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 March 2013
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SAC '13
Sponsor:
Acceptance Rates
SAC '13 Paper Acceptance Rate 255 of 1,063 submissions, 24%;
Overall Acceptance Rate 1,650 of 6,669 submissions, 25%
Upcoming Conference
SAC '25
- Sponsor:
- sigapp
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 85Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in