More Web Proxy on the site http://driver.im/

article

Metallaxis-FL: mutation-based fault localization

Authors:

Mike Papadakis,

Yves Le TraonAuthors Info & Claims

Software Testing, Verification & Reliability, Volume 25, Issue 5-7

Pages 605 - 628

https://doi.org/10.1002/stvr.1509

Published: 01 August 2015 Publication History

Abstract

Fault localization methods seek to identify faulty program statements based on the information provided by the failing and passing test executions. Spectrum-based methods are among the most popular ones and assist programmers by assigning suspiciousness values on program statements according to their probability of being faulty. This paper proposes Metallaxis, a fault localization approach based on mutation analysis. The innovative part of Metallaxis is that it uses mutants and links them with the faulty program places. Thus, mutants that are killed mostly by failing tests provide a good indication about the location of a fault. Experimentation using Metallaxis suggests that it is significantly more effective than statement-based approaches. This is true even in the case where mutation cost-reduction techniques, such as mutant sampling, are facilitated. Additionally, results from a controlled experiment show that the use of mutation as a testing technique provides benefits to the fault localization process. Therefore, fault localization is significantly improved by using mutation-based tests instead of block-based or branch-based test suites. Finally, evidence in support of the methods' scalability is also given. Copyright © 2013 John Wiley & Sons, Ltd.

References

[1]

Andrews JH, Briand LC, Labiche Y, Namin AS. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering 2006; Volume 32: pp.608-624.

[2]

Andrews JH, Briand LC, Labiche Y. Is mutation an appropriate tool for testing experiments? Proceedings of the 27th International Conference on Software Engineering, St. Louis, MO, USA, 2005; pp.402-411.

Digital Library

[3]

Zhang X, Guptam N, Gupta R. Pruning dynamic slices with confidence. SIGPLAN Notices 2006; Volume 41: pp.169-180.

[4]

Cleve H, Zeller A. Locating causes of program failures. Proceedings of the 27th International Conference on Software Engineering, St. Louis, MO, USA, 2005; pp.342-351.

Digital Library

[5]

Zeller A. Isolating cause-effect chains from computer programs. Proceedings of the 10th ACM SIGSOFT Symposium on Foundations of Software Engineering, Charleston, South Carolina, USA, 2002; pp.1-10.

Digital Library

[6]

Zeller A, Hildebrandt R. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering 2002; Volume 28: pp.183-200.

[7]

Wong WE, Debroy V, Choi B. A family of code coverage-based heuristics for effective fault localization. Journal of Systems and Software 2010; Volume 83: pp.188-208.

[8]

Santelices R, Jones JA, Yu Y, Harrold MJ. Lightweight fault-localization using multiple coverage types. Proceedings of the 31st International Conference on Software Engineering, Vancouver, BC, 2009; pp.56-66.

Digital Library

[9]

Abreu R, Zoeteweij P, Gemund AJCv. On the accuracy of spectrum-based fault localization. Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION, Windsor, 2007; pp.89-98.

[10]

Jones JA, Harrold MJ, Stasko J. Visualization of test information to assist fault localization. Software Engineering, 2002. ICSE 2002. Proceedings of the 24rd International Conference on, Orlando, FL, USA, 2002; pp.467-477.

Digital Library

[11]

Jones JA, Harrold MJ. Empirical evaluation of the tarantula automatic fault-localization technique. Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, Long Beach, CA, USA, 2005; pp.273-282.

Digital Library

[12]

Liu C, Yan X, Fei L, Han J, Midkiff SP. SOBER: Statistical model-based bug localization. SIGSOFT Software Engineering Notes 2005; Volume 30: pp.286-295.

[13]

Masri W. Fault localization based on information flow coverage. Software Testing, Verification and Reliability 2010; Volume 20: pp.121-147.

[14]

Yu K, Lin M, Gao Q, Zhang H, Zhang X. Locating faults using multiple spectra-specific models. Proceedings of the 2011 ACM Symposium on Applied Computing, TaiChung, Taiwan, 2011; pp.1404-1410.

Digital Library

[15]

Ali S, Andrews JH, Dhandapani T, Wang W. Evaluating the accuracy of fault localization techniques. Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, Auckland, New Zealand, 2009; pp.76-87.

Digital Library

[16]

Jia Y, Harman M. Higher order mutation testing. Information and Software Technology 2009; Volume 51: pp.1379-1393.

[17]

Jia Y, Harman M. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering 2011; Volume 37: pp.649-678.

[18]

Schuler D, Zeller A. Un-covering equivalent mutants. Software Testing, Verification and Validation ICST, 2010 Third International Conference on, Paris, 2010; pp.45-54.

Digital Library

[19]

Yue Y, Harman M. MILU: A customizable, runtime-optimized higher order mutation testing tool for the full C language. Practice and Research Techniques, 2008. TAIC PART '08. Testing: Academic & Industrial Conference, Windsor, 2008; pp.94-98.

Digital Library

[20]

Schuler D, Zeller A. Javalanche: Efficient mutation testing for Java. Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering ESEC/FSE '09, The Netherlands, 2009; pp.297-298.

Digital Library

[21]

Baudry B, Fleurey F, Traon YL. Improving test suites for efficient fault localization. Proceedings of the 28th International Conference on Software Engineering, Shanghai, China, 2006; pp.82-91.

Digital Library

[22]

Papadakis M, Le Traon Y. Using mutants to locate "unknown" faults. Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, Montreal, QC, 2012; pp.691-700.

Digital Library

[23]

DeMillo RA, Lipton RJ, Sayward FG. Hints on test data selection: help for the practicing programmer. Computer 1978; Volume 11: pp.34-41.

[24]

Offutt AJ, Untch RH. Mutation 2000: Uniting the orthogonal. In Mutation Testing for the New Century. Kluwer Academic Publishers: Norwell, MA, USA, 2001; pp.34-44. ISBN:0-7923-7323-5.

[25]

Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology 1996; Volume 5: pp.99-118.

[26]

Papadakis M, Malevris N. An empirical evaluation of the first and second order mutation testing strategies. Software Testing, Verification, and Validation Workshops ICSTW, 2010 Third International Conference on, Paris, 2010; pp.90-99.

Digital Library

[27]

Jeffrey D, Gupta N, Gupta R. Fault localization using value replacement. Proceedings of the 2008 International Symposium on Software Testing and Analysis, Seattle, WA, USA, 2008; pp.167-178.

Digital Library

[28]

Do H, Elbaum S, Rothermel G. Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empirical Software Engineering 2005; Volume 10: pp.405-435.

[29]

Hutchins M, Foster H, Goradia T, Ostrand T. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. Proceedings of the 16th International Conference on Software Engineering, Sorrento, Italy, 1994; pp.191-200.

[30]

Harder M, Mellen J, Ernst MD. Improving test suites via operational abstraction. Proceedings of the 25th International Conference on Software Engineering, Portland, Oregon, 2003; pp.60-71.

[31]

Yoo S, Harman M, Clark D. Fault localization prioritization: comparing information theoretic and coverage based approaches. ACM Transactions on Software Engineering Methodology 2013; Volume 22: pp.1-29.

[32]

Baah GK, Podgurski A, Harrold MJ. The probabilistic program dependence graph and its application to fault diagnosis. IEEE Transactions on Software Engineering 2010; Volume 36: pp.528-545.

[33]

Maldonado JC, Delamaro ME, Fabbri SCPF, Simão AdS, Sugeta T, Vincenzi AMR, Masiero PC. Proteum: A family of tools to support specification and program testing based on mutation. In Mutation Testing for the New Century, Wong WE ed. Kluwer Academic Publishers: Norwell, MA, USA, 2001; pp.113-116. ISBN: 0-7923-7323-5.

[34]

Agrawal H, DeMillo RA, Hathaway B, Hsu W, Hsu W, Krauser EW, Martin RJ, Mathur AP, Spafford E. Design of Mutant Operators for the C Programming Language. Purdue University: West Lafayette, Indiana, March 1989.

[35]

Zhang X, Gupta R. Whole execution traces and their applications. ACM Transactions on Architecture and Code Optimization 2005; Volume 2: pp.301-334.

[36]

Lyu MR, Horgan JR, London S. A coverage analysis tool for the effectiveness of software testing. Software Reliability Engineering, 1993. Proceedings., Fourth International Symposium on, Denver, CO, 1993; pp.25-34.

[37]

Fraser G, Zeller A. Mutation-driven generation of unit tests and oracles. Proceedings of the 19th International Symposium on Software Testing and Analysis, Trento, Italy, 2010; pp.147-158.

Digital Library

[38]

Kintis M, Papadakis M, Malevris N. Isolating first order equivalent mutants via second order mutation. Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, Montreal, QC, 2012; pp.701-710.

Digital Library

[39]

Parnin C, Orso A. Are automated debugging techniques actually helping programmers? Proceedings of the 2011 International Symposium on Software Testing and Analysis, Toronto, Ontario, Canada, 2011; pp.199-209.

Digital Library

[40]

Abreu R, Zoeteweij P, Gemund AJCv. Spectrum-based multiple fault localization. Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, Auckland, 2009; pp.88-99.

Digital Library

[41]

Burger M, Zeller A. Minimizing reproduction of software failures. Proceedings of the 2011 International Symposium on Software Testing and Analysis, Toronto, Ontario, Canada, 2011; pp.221-231.

Digital Library

[42]

Debroy V, Wong WE. Using mutation to automatically suggest fixes for faulty programs. Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation, Paris, 2010; pp.65-74.

Digital Library

[43]

Nica M, Nica S, Wotawa F. On the use of mutations and testing for debugging. Software: Practice and Experience 2013; Volume 43: pp.1121-1142.

[44]

Papadakis M, Malevris N. Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing. Software Quality Journal 2011; Volume 19: pp.691-723.

Digital Library

[45]

Papadakis M, Malevris N. Mutation based test case generation via a path selection strategy. Information and Software Technology 2012; Volume 54: pp.915-932.

[46]

Papadakis M, Malevris N. Automatic mutation test case generation via dynamic symbolic execution. Proceedings of the 2010 IEEE 21st International Symposium on Software Reliability Engineering ISSRE '10, San Jose, CA, 2010; pp.121-130.

Digital Library

Cited By

Zhang XSong YXie XXin QXing CFilkov VRay BZhou M(2024)Do not neglect what's on your hands: localizing software faults with exception trigger streamProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695479(982-994)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695479
Liu YZhu MDong JYu JHao DFilkov VRay BZhou M(2024)Compiler Bug Isolation via Enhanced Test Program MutationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695074(819-830)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695074
Rafi MKim DChen AChen TWang S(2024)Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code RepresentationProceedings of the ACM on Software Engineering10.1145/36607931:FSE(1937-1959)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660793
Show More Cited By

Metallaxis-FL: mutation-based fault localization
1. Computer systems organization
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Using Mutants to Locate "Unknown" Faults
ICST '12: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

Many fault localization techniques operate by crosscutting coverage information of passed and failed test executions. Generally, their accuracy depends on the utilized coverage elements and on the selected test cases. This paper proposes a novel fault ...
On similarity-awareness in testing-based fault localization

In the process of software development and maintenance, software debugging is an inevitable and time-consuming task. To accelerate software debugging, various approaches have been proposed to automate fault localization. Among them, testing-based fault-...
Fault density, fault types, and spectra-based fault localization

This paper presents multiple empirical experiments that investigate the impact of fault quantity and fault type on statistical, coverage-based fault localization techniques and fault-localization interference. Fault-localization interference is a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Software Testing, Verification & Reliability

Software Testing, Verification & Reliability Volume 25, Issue 5-7

August 2015

251 pages

ISSN:0960-0833

EISSN:1099-1689

Issue’s Table of Contents

Publisher

John Wiley and Sons Ltd.

United Kingdom

Publication History

Published: 01 August 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

103
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang XSong YXie XXin QXing CFilkov VRay BZhou M(2024)Do not neglect what's on your hands: localizing software faults with exception trigger streamProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695479(982-994)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695479
Liu YZhu MDong JYu JHao DFilkov VRay BZhou M(2024)Compiler Bug Isolation via Enhanced Test Program MutationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695074(819-830)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695074
Rafi MKim DChen AChen TWang S(2024)Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code RepresentationProceedings of the ACM on Software Engineering10.1145/36607931:FSE(1937-1959)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660793
Kang SAn GYoo S(2024)A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault LocalizationProceedings of the ACM on Software Engineering10.1145/36607711:FSE(1424-1446)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660771
Wu JZhang ZYang DXu JHe JMao X(2024)Knowledge-Augmented Mutation-Based Bug Localization for Hardware Design CodeACM Transactions on Architecture and Code Optimization10.1145/366052621:3(1-26)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3660526
Wang XYu HMeng XCao HZhang HSun HLiu XHu C(2024)MTL-TRANSFER: Leveraging Multi-task Learning and Transferred Knowledge for Improving Fault Localization and Program RepairACM Transactions on Software Engineering and Methodology10.1145/365444133:6(1-31)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3654441
Tian ZShu HWang DCao XKamei YChen JChristakis MPradel M(2024)Large Language Models for Equivalent Mutant Detection: How Far Are We?Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680395(1733-1745)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680395
Ouyang YYang JZhang LChristakis MPradel M(2024)Benchmarking Automated Program Repair: An Extensive Study on Both Real-World and Artificial BugsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652140(440-452)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3652140
Wu ZWu JZhang HLi ZChen JZheng ZXia QFan GZhen YChristakis MPradel M(2024)DAppFL: Just-in-Time Fault Localization for Decentralized Applications in Web3Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652116(137-148)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3652116
Guo HChen YChen XHuang YZheng Z(2024)Smart Contract Code Repair Recommendation based on Reinforcement Learning and Multi-metric OptimizationACM Transactions on Software Engineering and Methodology10.1145/363722933:4(1-31)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3637229
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents