More Web Proxy on the site http://driver.im/

research-article

An empirical study on the application of mutation testing for a safety-critical industrial software system

Authors:

Thomas Wetzlmaier,

Claus KlammerAuthors Info & Claims

SAC '17: Proceedings of the Symposium on Applied Computing

Pages 1401 - 1408

https://doi.org/10.1145/3019612.3019830

Published: 03 April 2017 Publication History

Abstract

Background: Testing is an essential activity in safety-critical software development, following high standards in terms of code coverage. Mutation testing allows assessing the effectiveness of testing and helps to further improve test cases. However, mutation testing is not widely practiced due to scalability problems when applied to real-world systems. Objective: The objective of the study is to investigate the applicability and usefulness of mutation testing for improving the quality of unit testing in context of safety-critical software systems. Method: A case study has been conducted together with an engineering company developing safety-critical systems. Mutation analysis has been applied to the studied system under test (60,000 LOC of C code) producing 75,043 mutants of which 27,158 survived test execution. A sample of 200 live mutants has been reviewed by the engineers, who also improved the existing unit test suite based on their findings. Findings: The reviewed sample contained 24+ equivalent mutants and 12+ duplicated mutants. It revealed a weak spot in the testing approach and provided valuable guidance to improve the existing unit test suite. Two new faults were found in the code when improving the tests. Test execution against the mutants required over 4,000 hours computing time. The overall effort was about half a person year.

References

[1]

Agrawal, H. et al. Design of mutant operators for the C programming language. Technical Report SERC-TR-41-P, Department of Computer Science, Purdue University, 1989.

[2]

Andrews, J.H., Briand, L.C., and Labiche, Y. Is Mutation an Appropriate Tool for Testing Experiments? In Proc. of 27th Int. Conf. on Software Engineering (ICSE), 2005, 402--411.

Digital Library

[3]

Baker, R. and Habli, I. An Empirical Evaluation of Mutation Testing for Improving the Test Quality of Safety-Critical Software. IEEE Transactions on Software Engineering. 39, 6 (June 2013), 787--805.

Digital Library

[4]

Flint, W. and Gilchrist, I. Software Testing and IEC 61508 - Project Case Study and Further Thoughts. In Safety-Critical Systems: Problems, Process and Practice. Springer London, 2009, 211--221.

[5]

Gopinath, R., Ahmed, I., Alipour, M.A., Jensen, C., and Groce, A. Does choice of mutation tool matter?. Software Quality Journal, to appear, online: May 2016.

[6]

Gopinath, R., Alipour, A., Ahmed, I., et al. How hard does mutation analysis have to be, anyway?. In Proc. of 26th Int. Symp. on Software Reliability Eng. (ISSRE), 2015, 216--227.

Digital Library

[7]

IEC. Standards 61508, Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems. International Electrotechnical Commission (IEC), 2010.

[8]

Jia, Y., and Harman, M. MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language. In Proc. of Testing: Academic Industrial Conference Practice and Research Techniques, 2008, 94--98.

Digital Library

[9]

Jia, Y., and Harman, M. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering. 37, 5 (Sep. 2011), 649--678.

Digital Library

[10]

Nica, S., Wotawa, F., and Ramler, R. Is Mutation Testing Scalable for Real-World Software Projects? In Proc. of 3rd Int. Conf. on Advances in System Testing and Validation Lifecycle (VALID). 2011, 40--45.

[11]

Offutt, A.J. Investigations of the Software Testing Coupling Effect. ACM Transactions on Software Engineering and Methodology, 1, 1 (Jan. 1992), 5--20.

Digital Library

[12]

Papadakis, M., Jia, Y., Harman, M., and Le Traon, Y. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In Proc. of 37th Int. Conf. on Software Engineering (ICSE), 2015, 936--946.

Digital Library

[13]

Runeson, P. and Höst, M. Guidelines for Conducting and Reporting Case Study Research in Software Engineering. Empirical Software Engineering. 14, 2 (Apr. 2009), 131--164.

Digital Library

[14]

Schuler, D. and Zeller, A. (Un-)Covering Equivalent Mutants. In Proc. of 3rd Int. Conf. on Software Testing, Verification and Validation (ICST), 2010, 45--54.

Digital Library

[15]

Smith, D.J., and Simpson, K.G.L. Safety Critical Systems Handbook: A Straight forward Guide to Functional Safety, IEC 61508 and Related Standards. Elsevier Science, 2010.

[16]

Thompson, S.K. Sampling. Wiley, 2012.

[17]

Zhu, H., Hall, P.A., and May, J.H. Software Unit Test Coverage and Adequacy. ACM Computing Surveys, 29, 4 (Dec. 1997), 366--427.

Digital Library

Cited By

Parsa SZakeri-Nasrabadi MTurhan B(2025)Testability-driven developmentComputer Standards & Interfaces10.1016/j.csi.2024.10387791:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.csi.2024.103877
Golmohammadi RParsa SZakeri-Nasrabadi M(2024)Dynamic domain testing with multi-agent Markov chain Monte Carlo methodSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-024-09680-528:13-14(8293-8317)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00500-024-09680-5
Viganò ECornejo OPastore FBriand L(2023)Data-Driven Mutation Analysis for Cyber-Physical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2022.321304149:4(2182-2201)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3213041
Show More Cited By

Index Terms

An empirical study on the application of mutation testing for a safety-critical industrial software system
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Extra-functional properties
      1. Software safety

Recommendations

Mutation-driven generation of unit tests and oracles
ISSTA '10: Proceedings of the 19th international symposium on Software testing and analysis

To assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a non-detected mutation indicates a weakness in the test suite. We present an automated approach to generate unit tests that detect these ...
Should software testers use mutation analysis to augment a test set?

Mutation testing has historically been used to assess the fault-finding effectiveness of a test suite or other verification technique. Mutation analysis, rather, entails augmenting a test suite to detect all killable mutants. Concerns about the time ...
On guiding the augmentation of an automated test suite via mutation analysis

Mutation testing has traditionally been used as a defect injection technique to assess the effectiveness of a test suite as represented by a "mutation score." Recently, mutation testing tools have become more efficient, and industrial usage of mutation ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '17: Proceedings of the Symposium on Applied Computing

April 2017

2004 pages

ISBN:9781450344869

DOI:10.1145/3019612

Conference Chair:
Sung Y. Shin
South Dakota State University
,
Program Chairs:
Dongwan Shin
New Mexico Tech
,
Maria Lencastre
University of Pernambuco, Brazil

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC 2017

Sponsor:

SIGAPP

SAC 2017: Symposium on Applied Computing

April 3 - 7, 2017

Marrakech, Morocco

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
275
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Parsa SZakeri-Nasrabadi MTurhan B(2025)Testability-driven developmentComputer Standards & Interfaces10.1016/j.csi.2024.10387791:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.csi.2024.103877
Golmohammadi RParsa SZakeri-Nasrabadi M(2024)Dynamic domain testing with multi-agent Markov chain Monte Carlo methodSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-024-09680-528:13-14(8293-8317)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00500-024-09680-5
Viganò ECornejo OPastore FBriand L(2023)Data-Driven Mutation Analysis for Cyber-Physical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2022.321304149:4(2182-2201)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3213041
Vercacmmen SBorg MDemeyer S(2023)Validation of Mutation Testing in the Safety Critical Industry through a Pilot Study2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00064(334-343)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00064
Örgård JGay Gde Oliveira Neto FViggedal K(2023)Mutation Testing in Continuous Integration: An Exploratory Industrial Case Study2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00063(324-333)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00063
Bartocci EMariani LNičković DYadav D(2023)Property-Based Mutation Testing2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00029(222-233)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00029
Lee JViganò ECornejo OPastore FBriand L(2023)Fuzzing for CPS Mutation Testing2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00079(1377-1389)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00079
Vercammen SDemeyer SBorg MPettersson NHedin G(2023)Mutation testing optimisations using the Clang front‐endSoftware Testing, Verification and Reliability10.1002/stvr.1865Online publication date: 17-Oct-2023
https://doi.org/10.1002/stvr.1865
van Heugten Breurkes JGilson FGalster M(2022)Overlap between Automated Unit and Acceptance Testing – a Systematic Literature ReviewProceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering10.1145/3530019.3530028(80-89)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3530019.3530028
Cornejo OPastore FBriand L(2022)Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space DomainIEEE Transactions on Software Engineering10.1109/TSE.2021.310768048:10(3913-3939)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TSE.2021.3107680
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents