More Web Proxy on the site http://driver.im/

short-paper

Learning for test prioritization: an industrial case study

Authors:

Benjamin Busjaeger,

Tao XieAuthors Info & Claims

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 975 - 980

https://doi.org/10.1145/2950290.2983954

Published: 01 November 2016 Publication History

Abstract

Modern cloud-software providers, such as Salesforce.com, increasingly adopt large-scale continuous integration environments. In such environments, assuring high developer productivity is strongly dependent on conducting testing efficiently and effectively. Specifically, to shorten feedback cycles, test prioritization is popularly used as an optimization mechanism for ranking tests to run by their likelihood of revealing failures. To apply test prioritization in industrial environments, we present a novel approach (tailored for practical applicability) that integrates multiple existing techniques via a systematic framework of machine learning to rank. Our initial empirical evaluation on a large real-world dataset from Salesforce.com shows that our approach significantly outperforms existing individual techniques.

References

[1]

R. Carlson, H. Do, and A. Denton. A clustering approach to improving test case prioritization: An industrial case study. In Proc ICSM 2011, pages 382–391.

Digital Library

[2]

J. Czerwonka, R. Das, N. Nagappan, A. Tarvo, and A. Teterev. CRANE: Failure prediction, change analysis and test prioritization in practice – experiences from Windows. In Proc. ICST 2011, pages 357–366.

Digital Library

[3]

H. Do, G. Rothermel, and A. Kinneer. Empirical studies of test case prioritization in a JUnit testing environment. In Proc. ISSRE 2004, pages 113–124.

Digital Library

[4]

S. Elbaum, D. Gable, and G. Rothermel. The impact of software evolution on code coverage information. In Proc. ICSM 2001, pages 170–179.

Digital Library

[5]

S. Elbaum, G. Rothermel, and J. Penix. Techniques for improving regression testing in continuous integration development environments. In Proc. FSE 2014, pages 235–245.

Digital Library

[6]

J.-M. Kim and A. Porter. A history-based test prioritization technique for regression testing in resource constrained environments. In Proc. ICSE 2002, pages 119–129.

Digital Library

[7]

H. Li. Learning to Rank for Information Retrieval and Natural Language Processing, Second Edition. Morgan & Claypool Publishers, 2014.

Digital Library

[8]

D. Marijan, A. Gotlieb, and S. Sen. Test case prioritization for continuous regression testing: An industrial case study. In Proc. ICSM 2013, pages 540–543.

Digital Library

[9]

A. Nanda, S. Mani, S. Sinha, M. J. Harrold, and A. Orso. Regression testing in the presence of non-code changes. In Proc. ICST 2011, pages 21–30.

Digital Library

[10]

T. B. Noor and H. Hemmati. A similarity-based approach for test case prioritization using historical failure data. In Proc. ISSRE 2015, pages 58–68.

Digital Library

[11]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: An empirical study. In Proc. ICSM 1999, pages 179–188.

Digital Library

[12]

R. Saha, L. Zhang, S. Khurshid, and D. Perry. An information retrieval approach for regression test prioritization based on program changes. In Proc. ICSE 2015, pages 268–279.

Digital Library

[13]

A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In Proc. ISSTA 2002, pages 97–106.

Digital Library

[14]

M. Tan, L. Tan, S. Dara, and C. Mayeux. Online defect prediction for imbalanced data. In Proc. ICSE 2015, pages 99–108.

Digital Library

[15]

C. D. Weissman and S. Bobrowski. The design of the force.com multitenant Internet application development platform. In Proc. SIGMOD 2009, pages 889–896.

Digital Library

[16]

S. Yoo, R. Nilsson, and M. Harman. Faster fault finding at Google using multi objective regression test optimisation. In Proc. ESEC/FSE 2011.

[17]

Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In Proc SIGIR 2007, pages 271–278.

Digital Library

Cited By

Su QLi XRen YQiu RHu CYin Y(2025)Attention Transfer Reinforcement Learning for Test Case Prioritization in Continuous IntegrationApplied Sciences10.3390/app1504224315:4(2243)Online publication date: 19-Feb-2025
https://doi.org/10.3390/app15042243
Tang PWang JLiu M(2025)Variational learning to rank for Test Case Prioritization via prioritizing metric inspired differentiable lossEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109776141(109776)Online publication date: Feb-2025
https://doi.org/10.1016/j.engappai.2024.109776
Kıran SEmre İTaşdelen S(2024)Prioritization of Regression Test Cases Based on Machine Learning MethodsGazi University Journal of Science10.35378/gujs.1446469(1-1)Online publication date: 29-Dec-2024
https://doi.org/10.35378/gujs.1446469
Show More Cited By

Index Terms

Learning for test prioritization: an industrial case study
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Optimizing test prioritization via test distribution analysis
ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Test prioritization aims to detect regression faults faster via reordering test executions, and a large number of test prioritization techniques have been proposed accordingly. However, test prioritization effectiveness is usually measured in terms of ...
Comparative study of machine learning test case prioritization for continuous integration testing
Abstract
There is a growing body of research indicating the potential of machine learning to tackle complex software testing challenges. One such challenge pertains to continuous integration testing, which is highly time-constrained, and generates a large ...
Model-based test prioritization heuristic methods and their evaluation
A-MOST '07: Proceedings of the 3rd international workshop on Advances in model-based testing

During regression testing, a modified system needs to be retested using the existing test suite. Since test suites may be very large, developers are interested in detecting faults in the system as early as possible. Test prioritization orders test cases ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2016

1156 pages

ISBN:9781450342186

DOI:10.1145/2950290

General Chair:
Thomas Zimmermann
Microsoft Research, USA
,
Program Chairs:
Jane Cleland-Huang
University of Notre Dame, USA
,
Zhendong Su
University of California at Davis, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

FSE'16

Sponsor:

SIGSOFT

FSE'16: 24nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering

November 13 - 18, 2016

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
866
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)3

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Su QLi XRen YQiu RHu CYin Y(2025)Attention Transfer Reinforcement Learning for Test Case Prioritization in Continuous IntegrationApplied Sciences10.3390/app1504224315:4(2243)Online publication date: 19-Feb-2025
https://doi.org/10.3390/app15042243
Tang PWang JLiu M(2025)Variational learning to rank for Test Case Prioritization via prioritizing metric inspired differentiable lossEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109776141(109776)Online publication date: Feb-2025
https://doi.org/10.1016/j.engappai.2024.109776
Kıran SEmre İTaşdelen S(2024)Prioritization of Regression Test Cases Based on Machine Learning MethodsGazi University Journal of Science10.35378/gujs.1446469(1-1)Online publication date: 29-Dec-2024
https://doi.org/10.35378/gujs.1446469
Wang GSun ZChen YZhao YLiang QHao DChristakis MPradel M(2024)Commit Artifact Preserving Build PredictionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680356(1236-1248)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680356
Felding EStrandberg PQuttineh NAfzal WHong JPark J(2024)Resource Constrained Test Case Prioritization with Simulated Annealing in an Industrial ContextProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing10.1145/3605098.3635971(1694-1701)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3605098.3635971
Corte RPietrantuono R(2024)Towards Log-driven Testing through Transformers: A Preliminary Study2024 19th European Dependable Computing Conference (EDCC)10.1109/EDCC61798.2024.00030(103-106)Online publication date: 8-Apr-2024
https://doi.org/10.1109/EDCC61798.2024.00030
Moubayed AAlhindawi NAlsakran JInjadat MKanan M(2024)A Data-Driven Approach Towards Software Regression Testing Quality Optimization2024 25th International Arab Conference on Information Technology (ACIT)10.1109/ACIT62805.2024.10877022(1-8)Online publication date: 10-Dec-2024
https://doi.org/10.1109/ACIT62805.2024.10877022
Khan MAzim ALiscano RSmith KChang YSeferi GTauseef Q(2024)On the Effectiveness of Feature Selection Techniques in the Context of ML-Based Regression Test PrioritizationIEEE Access10.1109/ACCESS.2024.345965612(131556-131575)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3459656
Zhang ZChen JGu YLi ZSosu R(2024)Exploiting DBSCAN and Combination Strategy to Prioritize the Test Suite in Regression TestingIET Software10.1049/2024/99429592024(1-14)Online publication date: 4-Apr-2024
https://doi.org/10.1049/2024/9942959
Roza EPrado Lima JVergilio S(2024)On the use of contextual information for machine learning based test case prioritization in continuous integration developmentInformation and Software Technology10.1016/j.infsof.2024.107444171:COnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107444
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten