Article

Automated Detection of Performance Regressions: The Mono Experience

Authors:

MASCOTS '05: Proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems

Pages 183 - 190

Published: 27 September 2005 Publication History

Publisher Site

Abstract

Engineering a large software project involves tracking the impact of development and maintenance changes on the software performance. An approach for tracking the impact is regression benchmarking, which involves automated benchmarking and evaluation of performance at regular intervals. Regression benchmarking must tackle the nondeterminism inherent to contemporary computer systems and execution environments and the impact of the nondeterminism on the results. On the example of a fully automated regression benchmarking environment for the Mono opensource project, we show how the problems associated with nondeterminism can be tackled using statistical methods.

Cited By

View all

Wu JDong JFang RZhang WWang WZuo DDwyer MDamian DZeller A(2022)FADATestProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510169(896-908)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510169
Bokhari MAlexander BWagner MCoello Coello C(2020)Towards rigorous validation of energy optimisation experimentsProceedings of the 2020 Genetic and Evolutionary Computation Conference10.1145/3377930.3390245(1232-1240)Online publication date: 25-Jun-2020
https://dl.acm.org/doi/10.1145/3377930.3390245
Mostafa SWang XXie TBultan TSen K(2017)PerfRanker: prioritization of performance regression tests for collection-intensive softwareProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092725(23-34)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092725
Show More Cited By

Index Terms

Recommendations

Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters
ICPE '15: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering

Performance testing is conducted before deploying system updates in order to ensure that the performance of large software systems did not degrade (i.e., no performance regressions). During such testing, thousands of performance counters are collected. ...
Automated detection of performance regressions using statistical process control techniques
ICPE '12: Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering

The goal of performance regression testing is to check for performance regressions in a new version of a software system. Performance regression testing is an important phase in the software development process. Performance regression testing is very ...
Automated root cause isolation of performance regressions during software development
ICPE '13: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering

Performance is crucial for the success of an application. To build responsive and cost efficient applications, software engineers must be able to detect and fix performance problems early in the development process. Existing approaches are either ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

MASCOTS '05: Proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems

September 2005

523 pages

ISBN:0769524583

Publisher

IEEE Computer Society

United States

Publication History

Published: 27 September 2005

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wu JDong JFang RZhang WWang WZuo DDwyer MDamian DZeller A(2022)FADATestProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510169(896-908)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510169
Bokhari MAlexander BWagner MCoello Coello C(2020)Towards rigorous validation of energy optimisation experimentsProceedings of the 2020 Genetic and Evolutionary Computation Conference10.1145/3377930.3390245(1232-1240)Online publication date: 25-Jun-2020
https://dl.acm.org/doi/10.1145/3377930.3390245
Mostafa SWang XXie TBultan TSen K(2017)PerfRanker: prioritization of performance regression tests for collection-intensive softwareProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092725(23-34)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092725
Bulej LBureš THorký VKotrăź JMarek LTrojánek TTźMa P(2017)Unit testing performance with Stochastic Performance LogicAutomated Software Engineering10.1007/s10515-015-0188-024:1(139-187)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s10515-015-0188-0
Luo QPoshyvanyk DGrechanik MKim MRobbes RBird C(2016)Mining performance regression inducing code changes in evolving softwareProceedings of the 13th International Conference on Mining Software Repositories10.1145/2901739.2901765(25-36)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2901739.2901765
Huang PMa XShen DZhou YJalote PBriand LHoek A(2014)Performance regression testing target prioritization via performance risk analysisProceedings of the 36th International Conference on Software Engineering10.1145/2568225.2568232(60-71)Online publication date: 31-May-2014
https://dl.acm.org/doi/10.1145/2568225.2568232
Lengauer PBitto VAngerer FGrünbacher PMössenböck HCollet PWąsowski AWeyer T(2014)Where has all my memory gone?Proceedings of the 8th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/2556624.2556628(1-8)Online publication date: 22-Jan-2014
https://dl.acm.org/doi/10.1145/2556624.2556628
Vitek JKalibera T(2012)R3ACM SIGPLAN Notices10.1145/2442776.244278147:4a(30-36)Online publication date: 18-Mar-2012
https://dl.acm.org/doi/10.1145/2442776.2442781
Vitek JKalibera TChakraborty SJerraya ABaruah SFischmeister S(2011)Repeatability, reproducibility, and rigor in systems researchProceedings of the ninth ACM international conference on Embedded software10.1145/2038642.2038650(33-38)Online publication date: 9-Oct-2011
https://dl.acm.org/doi/10.1145/2038642.2038650
Hocko MKalibera TAdamson ABondi AJuiz CSquillante M(2010)Reducing performance non-determinism via cache-aware page allocation strategiesProceedings of the first joint WOSP/SIPEW international conference on Performance engineering10.1145/1712605.1712640(223-234)Online publication date: 28-Jan-2010
https://dl.acm.org/doi/10.1145/1712605.1712640
Show More Cited By

Abstract

Cited By

Index Terms

Recommendations

Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters

Automated detection of performance regressions using statistical process control techniques

Automated root cause isolation of performance regressions during software development

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations