More Web Proxy on the site http://driver.im/

research-article

Open access

More Precise Regression Test Selection via Reasoning about Semantics-Modifying Changes

Authors:

Milos Gligoric,

Owolabi LegunsenAuthors Info & Claims

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 664 - 676

https://doi.org/10.1145/3597926.3598086

Published: 13 July 2023 Publication History

Abstract

Regression test selection (RTS) speeds up regression testing by only re-running tests that might be affected by code changes. Ideal RTS safely selects all affected tests and precisely selects only affected tests. But, aiming for this ideal is often slower than re-running all tests. So, recent RTS techniques use program analysis to trade precision for speed, i.e., lower regression testing time, or even use machine learning to trade safety for speed. We seek to make recent analysis-based RTS techniques more precise, to further speed up regression testing. Independent studies suggest that these techniques reached a “performance wall” in the speed-ups that they provide. We manually inspect code changes to discover those that do not require re-running tests that are only affected by such changes. We categorize 29 kinds of changes that we find from five projects into 13 findings, 11 of which are semantics-modifying. We enhance two RTS techniques---Ekstazi and STARTS---to reason about our findings. Using 1,150 versions of 23 projects, we evaluate the impact on safety and precision of leveraging such changes. We also evaluate if our findings from a few projects can speed up regression testing in other projects. The results show that our enhancements are effective and they can generalize. On average, they result in selecting 41.7% and 31.8% fewer tests, and take 33.7% and 28.7% less time than Ekstazi and STARTS, respectively, with no loss in safety.

References

[1]

2023. Apache Commons Codec. https://github.com/apache/commons-codec

[2]

2023. Apache Commons Email. https://github.com/apache/commons-email

[3]

2023. Apache Commons Math. https://github.com/apache/commons-math

[4]

2023. Data package for this paper. https://github.com/EngineeringSoftware/FineRTS

[5]

2023. Ekstazi. http://ekstazi.org/

[6]

2023. Gradle Enterprise Pricing. https://gradle.com/pricing/

[7]

2023. Gradle predictive test selection. https://gradle.com/gradle-enterprise-solutions/predictive-test-selection/

[8]

2023. HyRTS. http://hyrts.org

[9]

2023. STARTS—A tool for STAtic Regression Test Selection. https://github.com/TestingResearchIllinois/starts

[10]

Mohammed Nayef Al-Refai. 2019. Towards Model-Based Regression Test Selection. Ph. D. Dissertation. Colorado State University. USA.

[11]

Thomas Ball. 1998. On the Limit of Control Flow Analysis for Regression Test Selection. In International Symposium on Software Testing and Analysis. 134–142.

[12]

Jon Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. 2018. DeFlaker: Automatically Detecting Flaky Tests. In International Conference on Software Engineering. 433–444.

[13]

Antonia Bertolino, Antonio Guerriero, Breno Miranda, Roberto Pietrantuono, and Stefano Russo. 2020. Learning-to-Rank vs Ranking-to-Learn: Strategies for Regression Testing in Continuous Integration. In International Conference on Software Engineering. 1–12.

Digital Library

[14]

John Bible, Gregg Rothermel, and David S. Rosenblum. 2001. A Comparative Study of Coarse- and Fine-Grained Safe Regression Test-Selection Techniques. ACM Transactions on Software Engineering Methodology, 10, 2 (2001), 149–183.

Digital Library

[15]

David Binkley. 1997. Semantics Guided Regression Test Cost Reduction. IEEE Transactions on Software Engineering, 23, 8 (1997), 498–516.

Digital Library

[16]

David W Binkley. 1992. Using Semantic Differencing to Reduce the Cost of Regression Testing. In International Conference on Software Maintenance. 41–50.

[17]

Swarnendu Biswas, Rajib Mall, Manoranjan Satpathy, and Srihari Sukumaran. 2011. Regression Test Selection Techniques: A Survey. Informatica, 35, 3 (2011), 289–321.

[18]

Lionel Briand, Yvan Labiche, and Siyuan He. 2009. Automating Regression Test Selection Based on UML Designs. Journal of Information and Software Technology, 51, 1 (2009), 16–30.

Digital Library

[19]

Ahmet Celik, Young Chul Lee, and Milos Gligoric. 2018. Regression Test Selection for TizenRT. In International Symposium on Foundations of Software Engineering. 845–850.

[20]

Daniel Elsner, Florian Hauer, Alexander Pretschner, and Silke Reimer. 2021. Empirically Evaluating Readily Available Information for Regression Test Optimization in Continuous Integration. In International Symposium on Software Testing and Analysis. 491–504.

[21]

Emelie Engström, Per Runeson, and Mats Skoglund. 2010. A Systematic Review on Regression Test Selection Techniques. Journal of Information and Software Technology, 52, 1 (2010), 14–30.

Digital Library

[22]

Emelie Engström, Mats Skoglund, and Per Runeson. 2008. Empirical Evaluations of Regression Test Selection Techniques: A Systematic Review. In International Symposium on Empirical Software Engineering and Measurement. 22–31.

Digital Library

[23]

Beat Fluri, Michael Wursch, Martin PInzger, and Harald Gall. 2007. Change Distilling: Tree Differencing For Fine-Grained Source Code Change Extraction. IEEE Transactions on Software Engineering, 33, 11 (2007), 725–743.

Digital Library

[24]

Emanuel Giger, Martin Pinzger, and Harald C. Gall. 2011. Comparing Fine-Grained Source Code Changes and Code Churn for Bug Prediction. In Mining Software Repositories. 83–92.

[25]

Milos Gligoric. 2015. Regression Test Selection: Theory and Practice. Ph. D. Dissertation. University of Illinois at Urbana-Champaign. USA.

[26]

Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Ekstazi: Lightweight Test Selection. In International Conference on Software Engineering (Tool Demonstrations Track). 713–716.

[27]

Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Practical Regression Test Selection with Dynamic File Dependencies. In International Symposium on Software Testing and Analysis. 211–222.

[28]

Milos Gligoric, Rupak Majumdar, Rohan Sharma, Lamyaa Eloussi, and Darko Marinov. 2014. Regression Test Selection for Distributed Software Histories. In International Conference on Computer Aided Verification. 293–309

[29]

Milos Gligoric, Stas Negara, Owolabi Legunsen, and Darko Marinov. 2014. An Empirical Evaluation and Comparison of Manual and Automated Test Selection. In International Conference on Automated Software Engineering. 361–372.

[30]

Todd L. Graves, Mary Jean Harrold, Jung-Min Kim, Adam Porter, and Gregg Rothermel. 1998. An Empirical Study of Regression Test Selection Techniques. In International Conference on Software Engineering. 188–197.

[31]

Pooja Gupta, Mark Ivey, and John Penix. 2011. Testing at the speed and scale of Google. http://google-engtools.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html

[32]

Alex Gyori, Owolabi Legunsen, Farah Hariri, and Darko Marinov. 2018. Evaluating Regression Test Selection Opportunities in a Very Large Open-Source Ecosystem. In International Symposium on Software Reliability Engineering. 112–122.

[33]

Mary Jean Harrold, James A. Jones, Tongyu Li, Donglin Liang, Alessandro Orso, Maikel Pennings, Saurabh Sinha, S. Alexander Spoon, and Ashish Gujarathi. 2001. Regression Test Selection for Java Software. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 312–326.

[34]

Jean Hartmann. 2012. 30 Years of Regression Testing: Past, Present and Future. In Pacific Northwest Software Quality Conference. 119–126.

[35]

Ahmed E. Hassan. 2009. Predicting Faults using the Complexity of Code Changes. In International Conference on Software Engineering. 78–88.

Digital Library

[36]

René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In International Symposium on Software Testing and Analysis. 437–440.

Digital Library

[37]

D. Kung, J. Gao, P. Hsia, F. Wen, Y. Toyoshima, and C. Chen. 1994. Change Impact Identification in Object Oriented Software Maintenance. In International Conference on Software Maintenance. 202–211.

[38]

David Chenho Kung, Jerry Gao, Pei Hsia, Jeremy Lin, and Yasufumi Toyoshima. 1995. Class Firewall, Test Order, and Regression Testing of Object-Oriented Programs. Journal of Object-Oriented Programming, 8, 2 (1995), 51–65.

[39]

Owolabi Legunsen, Farah Hariri, August Shi, Yafeng Lu, Lingming Zhang, and Darko Marinov. 2016. An Extensive Study of Static Regression Test Selection in Modern Software Evolution. In International Symposium on Foundations of Software Engineering. 583–594.

[40]

Owolabi Legunsen, August Shi, and Darko Marinov. 2017. STARTS: STAtic Regression Test Selection. In International Conference on Automated Software Engineering. 949–954.

[41]

Hareton KN Leung and Lee White. 1989. Insights into Regression Testing. In International Conference on Software Maintenance. 60–69.

[42]

Hareton KN Leung and Lee White. 1991. A Cost Model to Compare Regression Test Strategies. In International Conference on Software Maintenance. 201–208.

[43]

Benjamin Livshits, John Whaley, and Monica S Lam. 2005. Reflection Analysis for Java. In Asian Symposium on Programming Languages and Systems. 139–160.

[44]

Erik Lundsten. 2019. EALRTS: A Predictive Regression Test Selection Tool. Master’s thesis. KTH Royal Institute of Technology. Sweden.

[45]

Mateusz Machalica, Alex Samylkin, Meredith Porth, and Satish Chandra. 2019. Predictive Test Selection. In International Conference on Software Engineering (Software Engineering in Practice). 91–100.

[46]

Matias Martinez and Martin Monperrus. 2015. Mining Software Repair Models for Reasoning on the Search Space of Automated Program Fixing. Empirical Software Engineering Journal, 20, 1 (2015), 176–205.

Digital Library

[47]

Raimund Moser, Witold Pedrycz, and Giancarlo Succi. 2008. A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction. In International Conference on Software Engineering. 181–190.

Digital Library

[48]

Nachiappan Nagappan and Thomas Ball. 2005. Use of Relative Code Churn Measures to Predict System Defect Density. In International Conference on Software Engineering. 284–292.

[49]

Hoan Anh Nguyen, Anh Tuan Nguyen, Tung Thanh Nguyen, Tien N. Nguyen, and Hridesh Rajan. 2013. A Study of Repetitiveness of Code Changes in Software Evolution. In International Conference on Automated Software Engineering. 180–190.

Digital Library

[50]

Alessandro Orso, Nanjuan Shi, and Mary Jean Harrold. 2004. Scaling Regression Testing to Large Software Systems. In International Symposium on Foundations of Software Engineering. 241–251.

Digital Library

[51]

Karl Palmskog, Ahmet Celik, and Milos Gligoric. 2020. Practical Machine-Checked Formalization of Change-Impact Analysis. In Tools and Algorithms for the Construction and Analysis of Systems. 137–157.

[52]

Rongqi Pan, Mojtaba Bagherzadeh, Taher A Ghaleb, and Lionel Briand. 2022. Test Case Selection and Prioritization using Machine Learning: A Systematic Literature Review. Empirical Software Engineering, 27, 2 (2022), 1–43.

Digital Library

[53]

Marek Parfianowicz. 2017. Open Clover. https://openclover.org

[54]

Xiaoxia Ren, Fenil Shah, Frank Tip, Barbara G Ryder, Ophelia Chesley, and Julian Dolby. 2003. Chianti: A Prototype Change Impact Analysis Tool for Java. Rutgers University CS Dept.

[55]

Gregg Rothermel and Mary Jean Harrold. 1993. A Safe, Efficient Algorithm for Regression Test Selection. In International Conference on Software Maintenance. 358–367.

[56]

Gregg Rothermel and Mary Jean Harrold. 1996. Analyzing Regression Test Selection Techniques. IEEE Transactions on Software Engineering, 22, 8 (1996), 529–551.

Digital Library

[57]

Gregg Rothermel and Mary Jean Harrold. 1997. A Safe, Efficient Regression Test Selection Technique. ACM Transactions on Software Engineering Methodology, 6, 2 (1997), 173–210.

Digital Library

[58]

Gregg Rothermel and Mary Jean Harrold. 1998. Empirical Studies of a Safe Regression Test Selection Technique. ACM Transactions on Software Engineering Methodology, 24, 6 (1998), 401–419.

Digital Library

[59]

Ripon K. Saha, Lingming Zhang, Sarfraz Khurshid, and Dewayne E. Perry. 2015. An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes. In International Conference on Software Engineering. 268–279.

[60]

August Shi, Milica Hadzi-Tanovic, Lingming Zhang, Darko Marinov, and Owolabi Legunsen. 2019. Reflection-Aware Static Regression Test Selection. In Conference on Object-Oriented Programming, Systems, Languages, and Applications. 187:1–187:29.

[61]

August Shi, Tifany Yung, Alex Gyori, and Darko Marinov. 2015. Comparing and Combining Test-Suite Reduction and Regression Test Selection. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 237–247.

[62]

August Shi, Peiyuan Zhao, and Darko Marinov. 2019. Understanding and Improving Regression Test Selection in Continuous Integration. In International Symposium on Software Reliability Engineering. 228–238.

[63]

Min Kyung Shin, Sudipto Ghosh, and Leo R Vijayasarathy. 2022. An Empirical Comparison of Four Java-based Regression Test Selection Techniques. Journal of Systems and Software, 186 (2022), 111174.

Digital Library

[64]

Mats Skoglund and Per Runeson. 2005. A Case Study of the Class Firewall Regression Test Selection Technique on a Large Scale Distributed Software System. In International Symposium on Empirical Software Engineering and Measurement. 74–83.

[65]

Mats Skoglund and Per Runeson. 2007. Improving Class Firewall Regression Test Selection by Removing the Class Firewall. International Journal on Software Engineering and Knowledge Engineering, 17, 3 (2007), 359–378.

[66]

Marko Vasic, Zuhair Parvez, Aleksandar Milicevic, and Milos Gligoric. 2017. File-Level vs. Module-Level Regression Test Selection for .NET. In International Symposium on Foundations of Software Engineering. 848–853.

Digital Library

[67]

Kaiyuan Wang, Chenguang Zhu, Ahmet Celik, Jongwook Kim, Don Batory, and Milos Gligoric. 2018. Towards Refactoring-Aware Regression Test Selection. In International Conference on Software Engineering. 233–244.

[68]

David Willmor and Suzanne M. Embury. 2005. A Safe Regression Test Selection Technique for Database Driven Applications. In International Conference on Software Maintenance. 421–430.

[69]

Guoqing Xu and Atanas Rountev. 2007. Regression Test Selection for AspectJ Software. In International Conference on Software Engineering. 65–74.

[70]

Nathan York. 2011. Tools for Continuous Integration at Google Scale. https://www.youtube.com/watch?v=b52aXZ2yi08

[71]

Maruf Hasan Zaber. 2021. Towards Parallelization of Regression Test Selection. Master’s thesis. University of California, Irvine. USA.

[72]

Jiyang Zhang, Yu Liu, Milos Gligoric, Owolabi Legunsen, and August Shi. 2022. Comparing and Combining Analysis-based and Learning-based Regression Test Selection. In ICSE Workshop on Automation of Software Test.

[73]

Lingming Zhang. 2018. Hybrid Regression Test Selection. In International Conference on Software Engineering. 199–209.

[74]

Jianjun Zhao, Tao Xie, and Nan Li. 2006. Towards Regression Test Selection for AspectJ Programs. In Workshop on Testing Aspect-Oriented Programs. 21–26.

[75]

Chenguang Zhu, Owolabi Legunsen, August Shi, and Milos Gligoric. 2019. A Framework for Checking Regression Test Selection Tools. In International Conference on Software Engineering. 430–441.

Digital Library

Cited By

Hui MWang LLi HYang RSong YZhuang HCui DLi Q(2025)Unveiling the microservices testing methods, challenges, solutions, and solutions gaps: A systematic mapping studyJournal of Systems and Software10.1016/j.jss.2024.112232220(112232)Online publication date: Feb-2025
https://doi.org/10.1016/j.jss.2024.112232
Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Li CBaz AShi AFilkov VRay BZhou M(2024)Reducing Test Runtime by Transforming Test FixturesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695541(1757-1769)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695541
Show More Cited By

Index Terms

More Precise Regression Test Selection via Reasoning about Semantics-Modifying Changes
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Software evolution
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Empirical Studies of a Safe Regression Test Selection Technique

Regression testing is an expensive testing procedure utilized to validate modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting a subset of a program's existing test suite. Safe regression ...
Comparing and combining test-suite reduction and regression test selection
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Regression testing is widely used to check that changes made to software do not break existing functionality, but regression test suites grow, and running them fully can become costly. Researchers have proposed test-suite reduction and regression test ...
Regression test selection on system requirements
ISEC '08: Proceedings of the 1st India software engineering conference

Regression testing, which is performed after changes are made to a software system, can be used before release of new versions of the system. However, practitioners often have little time to perform this regression testing because of the quick-release ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2023

1554 pages

ISBN:9798400702211

DOI:10.1145/3597926

General Chair:
René Just
University of Washington, USA
,
Program Chair:
Gordon Fraser
University of Passau, Germany

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '23

Sponsor:

SIGSOFT

ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 17 - 21, 2023

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
483
Total Downloads

Downloads (Last 12 months)324
Downloads (Last 6 weeks)27

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hui MWang LLi HYang RSong YZhuang HCui DLi Q(2025)Unveiling the microservices testing methods, challenges, solutions, and solutions gaps: A systematic mapping studyJournal of Systems and Software10.1016/j.jss.2024.112232220(112232)Online publication date: Feb-2025
https://doi.org/10.1016/j.jss.2024.112232
Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
Li CBaz AShi AFilkov VRay BZhou M(2024)Reducing Test Runtime by Transforming Test FixturesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695541(1757-1769)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695541
Zhang GLiu LChen ZWang JFilkov VRay BZhou M(2024)Hybrid Regression Test Selection by Integrating File and Method DependencesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695525(1557-1569)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695525
Liu LZhang GChen ZWang Jd'Amorim M(2024)Hybrid Regression Test Selection by Synergizing File and Method Call DependencesCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663872(669-670)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663872
Postolski IBraberman VGarbervetsky DUchitel Sd'Amorim M(2024)Verification of Programs with Common FragmentsCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663783(487-491)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663783
Lu JHou XWang HQiao XSong W(2024)A Multi-Granularity and Multi-Dimensional Code Knowledge Graph Construction Approach2024 6th International Conference on Communications, Information System and Computer Engineering (CISCE)10.1109/CISCE62493.2024.10653047(754-761)Online publication date: 10-May-2024
https://doi.org/10.1109/CISCE62493.2024.10653047
Liu YNie PGuo AGligoric MLegunsen OJust RFraser G(2023)Extracting Inline Tests from Unit TestsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598149(1458-1470)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598149
Yorihiro AJiang PMarqués VCarleton BLegunsen O(2023)eMOP: A Maven Plugin for Evolution-Aware Runtime VerificationRuntime Verification10.1007/978-3-031-44267-4_20(363-375)Online publication date: 3-Oct-2023
https://dl.acm.org/doi/10.1007/978-3-031-44267-4_20

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents