More Web Proxy on the site http://driver.im/

research-article

Leveraging mutants for automatic prediction of metamorphic relations using machine learning

Authors:

Sigrid EldhAuthors Info & Claims

MaLTeSQuE 2019: Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation

Pages 1 - 6

https://doi.org/10.1145/3340482.3342741

Published: 27 August 2019 Publication History

Abstract

An oracle is used in software testing to derive the verdict (pass/fail) for a test case. Lack of precise test oracles is one of the major problems in software testing which can hinder judgements about quality. Metamorphic testing is an emerging technique which solves both the oracle problem and the test case generation problem by testing special forms of software requirements known as metamorphic requirements. However, manually deriving the metamorphic requirements for a given program requires a high level of domain expertise, is labor intensive and error prone. As an alternative, we consider the problem of automatic detection of metamorphic requirements using machine learning (ML). For this problem we can apply graph kernels and support vector machines (SVM). A significant problem for any ML approach is to obtain a large labeled training set of data (in this case programs) that generalises well. The main contribution of this paper is a general method to generate large volumes of synthetic training data which can improve ML assisted detection of metamorphic requirements. For training data synthesis we adopt mutation testing techniques. This research is the first to explore the area of data augmentation techniques for ML-based analysis of software code. We also have the goal to enhance black-box testing using white-box methodologies. Our results show that the mutants incorporated into the source code corpus not only efficiently scale the dataset size, but they can also improve the accuracy of classification models.

References

[1]

Paul Ammann and Jeff Offutt. 2008. Introduction to software testing. Cambridge University Press.

Digital Library

[2]

Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering 41, 5 (2014), 507–525.

[3]

Andrew P Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7 (1997), 1145–1159.

Digital Library

[4]

Joshua Brown, Zhi Quan Zhou, and Yang-Wai Chow. 2018. Metamorphic testing of navigation software: A pilot study with Google Maps. In Proceedings of the 51st Hawaii International Conference on System Sciences.

[5]

Tsong Y Chen, Shing C Cheung, and Siu Ming Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. Technical Report. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong.

[6]

Tsong Yueh Chen, Fei-Ching Kuo, Wenjuan Ma, Willy Susilo, Dave Towey, Jeffrey Voas, and Zhi Quan Zhou. 2016. Metamorphic testing for cybersecurity. Computer 49, 6 (2016), 48–55.

Digital Library

[7]

Tsong Yueh Chen, Tsun Him Tse, and Z Quan Zhou. 2003. Fault-based testing without the need of oracles. Information and Software Technology 45, 1 (2003), 1–9.

Digital Library

[8]

N.J. Cutland. 1980. Computability: an Introduction to Recursive Function Theory. Cambridge University Press, Cambridge.

[9]

Arni Einarsson and Janus Dam Nielsen. 2008. A SurvivorâĂŹs guide to Java program analysis with soot. BRICS, Department of Computer Science, University of Aarhus, Denmark (2008), 17.

[10]

Robert Geist, A. Jefferson Offutt, and Frederick C Harris. 1992. Estimation and enhancement of real-time software reliability through mutation analysis. IEEE Trans. Comput. 41, 5 (1992), 550–558.

Digital Library

[11]

Bonnie Hardin and Upulee Kanewala. 2018. Using semi-supervised learning for predicting metamorphic relations. In Proceedings of the 3rd International Workshop on Metamorphic Testing. ACM, 14–17.

Digital Library

[12]

Darryl C Jarman, Zhi Quan Zhou, and Tsong Yueh Chen. 2017. Metamorphic testing for Adobe data analytics software. In Proceedings of the 2nd International Workshop on Metamorphic Testing. IEEE Press, 21–27.

Digital Library

[13]

Upulee Kanewala and James M Bieman. 2013. Using machine learning techniques to detect metamorphic relations for programs without test oracles. In 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 1–10.

[14]

Upulee Kanewala, James M Bieman, and Asa Ben-Hur. 2016. Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels. Software testing, verification and reliability 26, 3 (2016), 245–269.

Digital Library

[15]

Bob Kurtz, Paul Ammann, Jeff Offutt, and Mariet Kurtz. 2016. Are we there yet? How redundant and equivalent mutants affect determination of test completeness. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 142–151.

[16]

Mikael Lindvall, Dharmalingam Ganesan, Ragnar Árdal, and Robert E Wiegand. 2015. Metamorphic model-based testing applied on NASA DAT: an experience report. In Proceedings of the 37th International Conference on Software Engineering-Volume 2. IEEE Press, 129–138.

Digital Library

[17]

Christian Murphy, Gail E Kaiser, and Lifeng Hu. 2008. Properties of machine learning applications for use in metamorphic testing. (2008).

[18]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.

Digital Library

[19]

Karishma Rahman and Upulee Kanewala. 2018. Predicting metamorphic relations for matrix calculation programs. In Proceedings of the 3rd International Workshop on Metamorphic Testing. ACM, 10–13.

Digital Library

[20]

Raja Vallee-Rai and Laurie J Hendren. 1998. Jimple: Simplifying Java bytecode for analyses and transformations. (1998).

[21]

Elaine J Weyuker. 1982. On testing non-testable programs. Comput. J. 25, 4 (1982), 465–470.

[22]

Sebastien C Wong, Adam Gatt, Victor Stamatescu, and Mark D McDonnell. 2016. Understanding data augmentation for classification: when to warp?. In 2016 international conference on digital image computing: techniques and applications (DICTA). IEEE, 1–6. Abstract 1 Introduction 2 Background 2.1 Metamorphic Testing 3 Proposed Workflow 4 Implementation Results 4.1 Dataset Creation 4.2 Mutants for Source Code Data Augmentation 4.3 Effectiveness of the Mutant Incorporated Dataset 5 Related Work 6 Conclusion and Future Work Acknowledgments References

Cited By

Idialu OMathews NMaipradit RAtlee JNagappan MSpinellis DConstantinou EBacchelli A(2024)Whodunit: Classifying Code as Human Authored or GPT-4 Generated - A case study on CodeChef problemsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644926(394-406)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644926
Ayerdi JTerragni VJahangirova GArrieta ATonella P(2024)GenMorph: Automatically Generating Metamorphic Relations via Genetic ProgrammingIEEE Transactions on Software Engineering10.1109/TSE.2024.3407840(1-12)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3407840
Sharma TKechagia MGeorgiou STiwari RVats IMoazen HSarro F(2024)A survey on machine learning techniques applied to source codeJournal of Systems and Software10.1016/j.jss.2023.111934209:COnline publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111934
Show More Cited By

Index Terms

Leveraging mutants for automatic prediction of metamorphic relations using machine learning
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Kernel methods
        Support vector machines
2. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software reliability

Recommendations

Metamorphic Testing: A Review of Challenges and Opportunities

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and ...
Using semi-supervised learning for predicting metamorphic relations
MET '18: Proceedings of the 3rd International Workshop on Metamorphic Testing

Software testing is difficult to automate, especially in programs which have no oracle, or method of determining which output is correct. Metamorphic testing is a solution this problem. Metamorphic testing uses metamorphic relations to define test cases ...
An application of metamorphic testing for testing scientific software
MET '16: Proceedings of the 1st International Workshop on Metamorphic Testing

One of the grand challenges for adequately testing scientific software is due to the oracle problem. Metamorphic testing is the promise technique for addressing the problem through checking the satisfaction of a metamorphic relation that defines the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MaLTeSQuE 2019: Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation

August 2019

42 pages

ISBN:9781450368551

DOI:10.1145/3340482

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

EU ITEA3

Conference

ESEC/FSE '19

Sponsor:

SIGSOFT

ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 27, 2019

Tallinn, Estonia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
484
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)6

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Idialu OMathews NMaipradit RAtlee JNagappan MSpinellis DConstantinou EBacchelli A(2024)Whodunit: Classifying Code as Human Authored or GPT-4 Generated - A case study on CodeChef problemsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644926(394-406)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644926
Ayerdi JTerragni VJahangirova GArrieta ATonella P(2024)GenMorph: Automatically Generating Metamorphic Relations via Genetic ProgrammingIEEE Transactions on Software Engineering10.1109/TSE.2024.3407840(1-12)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3407840
Sharma TKechagia MGeorgiou STiwari RVats IMoazen HSarro F(2024)A survey on machine learning techniques applied to source codeJournal of Systems and Software10.1016/j.jss.2023.111934209:COnline publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111934
Zheng XJiang MQuan Zhou Z(2024)Boosting Metamorphic Relation Prediction via Code Representation Learning: An Empirical StudySoftware Testing, Verification and Reliability10.1002/stvr.1889Online publication date: 8-Jul-2024
https://doi.org/10.1002/stvr.1889
Islam MKhan FAlam SHasan M(2023)Artificial Intelligence in Software Testing: A Systematic ReviewTENCON 2023 - 2023 IEEE Region 10 Conference (TENCON)10.1109/TENCON58879.2023.10322349(524-529)Online publication date: 31-Oct-2023
https://doi.org/10.1109/TENCON58879.2023.10322349
Nasr INassar LKarray FZayed M(2023)Enhanced Deep Learning Satellite-based Model for Yield Forecasting and Quality Assurance Using Metamorphic Testing2023 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN54540.2023.10191179(1-6)Online publication date: 18-Jun-2023
https://doi.org/10.1109/IJCNN54540.2023.10191179
Fontes AGay G(2023)The integration of machine learning into automated test generation: A systematic mapping studySoftware Testing, Verification and Reliability10.1002/stvr.184533:4Online publication date: 2-May-2023
https://doi.org/10.1002/stvr.1845
Qiu KZheng ZChen TPoon P(2022)Theoretical and Empirical Analyses of the Effectiveness of Metamorphic Relation CompositionIEEE Transactions on Software Engineering10.1109/TSE.2020.300969848:3(1001-1017)Online publication date: 1-Mar-2022
https://doi.org/10.1109/TSE.2020.3009698
Duque-Torres APfahl DKlammer CFischer S(2022)Using Source Code Metrics for Predicting Metamorphic Relations at Method Level2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00132(1147-1154)Online publication date: Mar-2022
https://doi.org/10.1109/SANER53432.2022.00132
Duque-Torres APfahl D(2022)Inferring Metamorphic Relations from JavaDocs: A Deep Dive into the MeMo ApproachProduct-Focused Software Process Improvement10.1007/978-3-031-21388-5_29(418-432)Online publication date: 14-Nov-2022
https://doi.org/10.1007/978-3-031-21388-5_29
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents