Abstract
Performance is an important aspect of software quality. To prevent performance degradation during software development, performance can be monitored and software modifications that damage performance can be reverted or optimized. Regression benchmarking provides means for an automated monitoring of performance, yielding a list of software modifications potentially associated with performance changes. We focus on locating individual modifications as causes of individual performance changes and present three methods that help narrow down the list of modifications potentially associated with a performance change. We illustrate the entire process on a real world project.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Jeffries, R.E., Anderson, A., Hendrickson, C.: Extreme Programming Installed. Addison-Wesley Longman Publishing Co., Inc., Boston (2000)
Bulej, L., Kalibera, T., Tuma, P.: Repeated results analysis for middleware regression benchmarking. Performance Evaluation 60, 345–358 (2005)
Bulej, L., Kalibera, T., Tuma, P.: Regression benchmarking with simple middleware benchmarks. In: Hassanein, H., Olivier, R.L., Richard, G.G., Wilson, L.L. (eds.) International Workshop on Middleware Performance, IPCCC 2004, pp. 771–776 (2004)
Kalibera, T., Bulej, L., Tuma, P.: Generic environment for full automation of benchmarking. In: Beydeda, S., Gruhn, V., Mayer, J., Reussner, R., Schweiggert, F. (eds.) SOQUA/TECOS. LNI, GI vol. 58, pp. 125–132 (2004)
Novell, Inc.: The Mono Project (2005), http://www.mono-project.com
ECMA: ECMA-335: Common Language Infrastructure (CLI). ECMA (European Association for Standardizing Information and Communication Systems), Geneva, Switzerland (2002)
Distributed Systems Research Group: Mono regression benchmarking (2005), http://nenya.ms.mff.cuni.cz/projects/mono
DOC Group: TAO performance scoreboard (2005), http://www.dre.vanderbilt.edu/stats/performance.shtml
Prochazka, M., Madan, A., Vitek, J., Liu, W.: RTJBench: A Real-Time Java Benchmarking Framework. In: Component And Middleware Performance Workshop, OOPSLA 2004 (2004)
Dillenseger, B., Cecchet, E.: CLIF is a Load Injection Framework. In: Workshop on Middleware Benchmarking: Approaches, Results, Experiences, OOPSLA 2003 (2003)
Memon, A.M., Porter, A.A., Yilmaz, C., Nagarajan, A., Schmidt, D.C., Natarajan, B.: Skoll: Distributed continuous quality assurance. In: ICSE, pp. 459–468. IEEE Computer Society, Los Alamitos (2004)
Courson, M., Mink, A., Marçais, G., Traverse, B.: An automated benchmarking toolset. In: Bubak, M., Afsarmanesh, H., Williams, R., Hertzberger, L.O. (eds.) HPCN-Europe 2000. LNCS, vol. 1823, pp. 497–506. Springer, Heidelberg (2000)
Kalibera, T., Bulej, L., Tuma, P.: Benchmark precision and random initial state. In: accepted for 2005 International Symposium on Performance Evaluation of Computer and Telecommunications Systems, SPECTS 2005 (2005)
Buble, A., Bulej, L., Tuma, P.: CORBA benchmarking: A course with hidden obstacles. In: IPDPS, p. 279. IEEE Computer Society, Los Alamitos (2003)
Hauswirth, M., Sweeney, P., Diwan, A., Hind, M.: The need for a whole-system view of performance. In: Component And Middleware Performance workshop, OOPSLA 2004 (2004)
Gu, D., Verbrugge, C., Gagnon, E.: Code layout as a source of noise in JVM performance. In: Component And Middleware Performance Workshop, OOPSLA 2004 (2004)
The FreeBSD Documentation Project: FreeBSD Developers’ Handbook (2005), http://www.freebsd.org/doc/en/books/developers-handbook
Jain, R.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley-Interscience, New York (1991)
Re, C., Vogels, W.: SciMark – C# (2004), http://rotor.cs.cornell.edu/SciMark/
Pozo, R., Miller, B.: SciMark 2.0 benchmark (2005), http://math.nist.gov/scimark2/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kalibera, T., Bulej, L., Tuma, P. (2005). Quality Assurance in Performance: Evaluating Mono Benchmark Results. In: Reussner, R., Mayer, J., Stafford, J.A., Overhage, S., Becker, S., Schroeder, P.J. (eds) Quality of Software Architectures and Software Quality. QoSA SOQUA 2005 2005. Lecture Notes in Computer Science, vol 3712. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11558569_20
Download citation
DOI: https://doi.org/10.1007/11558569_20
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29033-9
Online ISBN: 978-3-540-32056-2
eBook Packages: Computer ScienceComputer Science (R0)