More Web Proxy on the site http://driver.im/

research-article

Improving the accuracy of oracle verdicts through automated model steering

Authors:

Sanjai Rayadurgam,

Mats P.E. HeimdahlAuthors Info & Claims

ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

Pages 527 - 538

https://doi.org/10.1145/2642937.2642989

Published: 15 September 2014 Publication History

Abstract

The oracle - a judge of the correctness of the system under test (SUT) - is a major component of the testing process. Specifying test oracles is challenging for some domains, such as real-time embedded systems, where small changes in timing or sensory input may cause large behavioral differences. Models of such systems, often built for analysis and simulation, are appealing for reuse as oracles. These models, however, typically represent an idealized system, abstracting away certain issues such as non-deterministic timing behavior and sensor noise. Thus, even with the same inputs, the model's behavior may fail to match an acceptable behavior of the SUT, leading to many false positives reported by the oracle.

We propose an automated steering framework that can adjust the behavior of the model to better match the behavior of the SUT to reduce the rate of false positives. This model steering is limited by a set of constraints (defining acceptable differences in behavior) and is based on a search process attempting to minimize a dissimilarity metric. This framework allows non-deterministic, but bounded, behavior differences, while preventing future mismatches, by guiding the oracle-within limits-to match the execution of the SUT. Results show that steering significantly increases SUT-oracle conformance with minimal masking of real faults and, thus, has significant potential for reducing false positives and, consequently, development costs.

References

[1]

IBM Rational Rhapsody. http://www.ibm.com/developerworks/rational/products/rhapsody/, 2014.

[2]

J. Andrews, L. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? Proc of the 27th Int'l Conf on Software Engineering (ICSE), pages 402--411, 2005.

Digital Library

[3]

J. Andrews, L. Briand, Y. Labiche, and A. Namin. Using mutation analysis for assessing and comparing testing coverage criteria. Software Engineering, IEEE Transactions on, 32(8):608--624, aug. 2006.

Digital Library

[4]

A. Arcuri, M. Z. Iqbal, and L. Briand. Black-box system testing of real-time embedded systems using random and search-based testing. In Proceedings of the 22nd IFIP WG 6.1 Int'l Conf. on Testing software and systems, pages 95--110. Springer-Verlag, 2010.

Digital Library

[5]

L. B. Briones and E. Brinksma. A test generation framework for quiescent real-time systems. In IN FATES'04, pages 64--78. Springer-Verlag GmbH, 2004.

Digital Library

[6]

S.-H. Cha. Comprehensive survey on distance/similarity measures between probability density functions. International Journal of Mathematical Models and Methods in Applied Sciences, 1(4):300--307, 2007.

[7]

D. Coppit and J. Haddox-Schatz. On the use of specification-based assertions as test oracles. In Proceedings of the 29th Annual IEEE/NASA on Software Engineering Workshop, SEW '05, pages 305--314, Washington, DC, USA, 2005. IEEE Computer Society.

Digital Library

[8]

L. De Moura and N. Bjørner. Z3: An efficient smt solver. In Tools and Algorithms for the Construction and Analysis of Systems, pages 337--340. Springer, 2008.

[9]

A. En-Nouaary, R. Dssouli, and F. Khendek. Timed wp-method: testing real-time systems. Software Engineering, IEEE Transactions on, 28(11):1023--1038, Nov.

Digital Library

[10]

G. Gay, S. Rayadurgam, and M. P. Heimdahl. Steering model-based oracles to admit real program behaviors. In Proceedings of the 36th International Conference on Software Engineering -- NIER Track, ICSE '14, New York, NY, USA, 2014. ACM.

Digital Library

[11]

A. Gomes, A. Mota, A. Sampaio, F. Ferri, and E. Watanabe. Constructive model-based analysis for safety assessment. Int'l Journal on Software Tools for Technology Transfer, 14(6):673--702, 2012.

Digital Library

[12]

W. Gu, J. Vetter, and K. Schwan. An annotated bibliography of interactive program steering. ACM SIGPLAN Notices, 29, 1994.

Digital Library

[13]

G. Hagen. Verifying safety properties of Lustre programs: an SMT-based approach. PhD thesis, University of Iowa, December 2008.

Digital Library

[14]

N. Halbwachs. Synchronous Programming of Reactive Systems. Klower Academic Press, 1993.

Digital Library

[15]

D. Harel, H. Lachover, A. Naamad, A. Pnueli, M. Politi, R. Sherman, A. Shtull-Trauring, and M. Trakhtenbrot. Statemate: A working environment for the development of complex reactive systems. IEEE Transactions on Software Engineering, 16(4):403--414, April 1990.

Digital Library

[16]

M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. A comprehensive survey of trends in oracles for software testing. Technical Report CS-13-01, University of Sheffield, Department of Computer Science, 2013.

[17]

W. Howden. Theoretical and empirical studies of program testing. IEEE Transactions on Software Engineering, 4(4):293--298, 1978.

Digital Library

[18]

M. S. Jaffe, N. G. Leveson, M. P. Heimdahl, and B. E. Melhart. Software requirements analysis for real-time process-control systems. IEEE Transactions on Software Engineering, 17(3):241--258, March 1991.

Digital Library

[19]

S. Kannan, M. Kim, I. Lee, O. Sokolsky, and M. Viswanathan. Run-time monitoring and steering based on formal specifications. In Workshop on Modeling Software System Structures in a Fastly Moving Scenario, 2000.

[20]

K. G. Larsen, M. Mikucionis, and B. Nielsen. Online testing of real-time systems using UPPAAL. In Int'l workshop on formal approaches to testing of software (FATES 04). Springer, 2004.

Digital Library

[21]

D. Lee and M. Yannakakis. Principles and methods of testing finite state machines-a survey. Proceedings of the IEEE, 84(8):1090--1123, 1996.

[22]

MathWorks Inc. Stateflow. http://www.mathworks.com/stateflow.

[23]

D. Miller, J. Guo, E. Kraemer, and Y. Xiong. On-the-fly calculation and verification of consistent steering transactions. In Supercomputing, ACM/IEEE 2001 Conf., pages 8--8, 2001.

Digital Library

[24]

S. P. Miller, A. C. Tribble, M. W. Whalen, and M. P. E. Heimdahl. Proving the shalls: Early validation of requirements through formal methods. Int. J. Softw. Tools Technol. Transf., 8(4):303--319, 2006.

Digital Library

[25]

A. Murugesan, S. Rayadurgam, and M. Heimdahl. Modes, features, and state-based modeling for clarity and flexibility. In Proceedings of the 2013 Workshop on Modeling in Software Engineering, 2013.

Digital Library

[26]

M. Pezze and M. Young. Software Test and Analysis: Process, Principles, and Techniques. John Wiley and Sons, October 2006.

Digital Library

[27]

A. Rajan, M. Whalen, and M. Heimdahl. The effect of program and model structure on MC/DC test adequacy coverage. In Proc. of the 30th Int'l Conf. on Software engineering, pages 161--170. ACM, 2008.

Digital Library

[28]

A. Rajan, M. Whalen, M. Staats, and M. Heimdahl. Requirements coverage as an adequacy measure for conformance testing, 2008.

Digital Library

[29]

T. Savor and R. Seviora. An approach to automatic detection of software failures in real-time systems. In Real-Time Technology and Applications Symposium, 1997. Proceedings., Third IEEE, pages 136--146, 1997.

[30]

M. Staats, G. Gay, and M. Heimdahl. Automated oracle creation support, or: how I learned to stop worrying about fault propagation and love mutation testing. In Proceedings of the 2012 Int'l Conf. on Software Engineering, pages 870--880. IEEE Press, 2012.

Digital Library

[31]

M. Veanes, C. Campbell, W. Grieskamp, W. Schulte, N. Tillmann, and L. Nachmanson. Model-based testing of object-oriented reactive systems with spec explorer. In R. M. Hierons, J. P. Bowen, and M. Harman, editors, Formal Methods and Testing, volume 4949 of Lecture Notes in Computer Science, pages 39--76. Springer, 2008.

Digital Library

[32]

E. Weyuker. The oracle assumption of program testing.

Cited By

Alonso JSegura SRuiz-Cortés AJust RFraser G(2023)AGORA: Automated Generation of Test Oracles for REST APIsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598114(1018-1030)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598114
Ibrahimzada AVarli YTekinoglu DJabbarvand RRoychoudhury ACadar CKim M(2022)Perfect is the enemy of test oracleProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549086(70-81)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549086
Jabbarvand RMehralian FMalek SDevanbu PCohen MZimmermann T(2020)Automated construction of energy test oracles for AndroidProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409677(927-938)Online publication date: 8-Nov-2020
https://dl.acm.org/doi/10.1145/3368089.3409677
Show More Cited By

Index Terms

Improving the accuracy of oracle verdicts through automated model steering
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Steering model-based oracles to admit real program behaviors
ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software Engineering

The oracle—an arbiter of correctness of the system under test (SUT)—is a major component of the testing process. Specifying oracles is particularly challenging for real-time embedded systems, where small changes in time or sensor inputs may cause large ...
What test oracle should i use for effective GUI testing?
ASE'03: Proceedings of the 18th IEEE International Conference on Automated Software Engineering

Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. In this paper we show that the test oracle used during testing also contributes ...
An orchestrated survey of methodologies for automated software test case generation

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

September 2014

934 pages

ISBN:9781450330138

DOI:10.1145/2642937

General Chair:
Ivica Crnkovic
Mälardalen University, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Paul Grünbacher
Johannes Kepler Universität Linz, Austria

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGSOFT: ACM Special Interest Group on Software Engineering
Mälardalen University: Mälardalen University
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Computer and Network Systems

Conference

ASE '14

Sponsor:

SIGAI
SIGSOFT
Mälardalen University

ASE '14: ACM/IEEE International Conference on Automated Software Engineering

September 15 - 19, 2014

Vasteras, Sweden

Acceptance Rates

ASE '14 Paper Acceptance Rate 82 of 337 submissions, 24%;

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
189
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alonso JSegura SRuiz-Cortés AJust RFraser G(2023)AGORA: Automated Generation of Test Oracles for REST APIsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598114(1018-1030)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598114
Ibrahimzada AVarli YTekinoglu DJabbarvand RRoychoudhury ACadar CKim M(2022)Perfect is the enemy of test oracleProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549086(70-81)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549086
Jabbarvand RMehralian FMalek SDevanbu PCohen MZimmermann T(2020)Automated construction of energy test oracles for AndroidProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409677(927-938)Online publication date: 8-Nov-2020
https://dl.acm.org/doi/10.1145/3368089.3409677
Gay GRayadurgam SHeimdahl M(2017)Automated Steering of Model-Based Test Oracles to Admit Real Program BehaviorsIEEE Transactions on Software Engineering10.1109/TSE.2016.261531143:6(531-555)Online publication date: 10-Jun-2017
https://dl.acm.org/doi/10.1109/TSE.2016.2615311
Barr EHarman MMcMinn PShahbaz MShin Yoo (2015)The Oracle Problem in Software Testing: A SurveyIEEE Transactions on Software Engineering10.1109/TSE.2014.237278541:5(507-525)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1109/TSE.2014.2372785
You DRayadurgam SHeimdahl MKomp JKim BSokolsky OCohen MGrunske LWhalen M(2015)Executing model-based tests on platform-specific implementationsProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2015.64(418-428)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1109/ASE.2015.64

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents