More Web Proxy on the site http://driver.im/

Article

Free access

Structural specification-based testing: automated support and experimental evaluation

Authors:

Debra J. RichardsonAuthors Info & Claims

ESEC/FSE-7: Proceedings of the 7th European software engineering conference held jointly with the 7th ACM SIGSOFT international symposium on Foundations of software engineering

Pages 285 - 302

Published: 01 October 1999 Publication History

Abstract

In this paper, we describe a testing technique, called structural specification-based testing (SST), which utilizes the formal specification of a program unit as the basis for test selection and test coverage measurement. We also describe an automated testing tool, called ADLscope, which supports SST for program units specified in Sun Microsystems' Assertion Definition Language (ADL). ADLscope automatically generates coverage conditions from a program's ADL specification. While the program is tested, ADLscope determines which of these conditions are covered by the tests. An uncovered condition exhibits aspects of the specification inadequately exercised during testing. The tester uses this information to develop new test data to exercise the uncovered conditions.

We provide an overview of SST's specification-based test criteria and describe the design and implementation of ADLscope. Specification-based testing is guided by a specification, whereby the testing activity is directly related to what a component under test is supposed to do, rather than what it actually does. Specification-based testing is a significant advance in testing, because it is often more straightforward to accomplish and it can reveal failures that are often missed by traditional code-based testing techniques. As an initial evaluation of the capabilities of specification-based testing, we conducted an experiment to measure defect detection capabilities, code coverage and usability of SST/ADLscope; we report here on the results.

References

[1]

A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Mutation analysis. Technical Report TR GIT-ICS-79/08, Georgia Institute of Technology, September 1979.]]

[2]

Bender and Associates. http://www.softtest.com.]]

[3]

T. A. Budd, R. J. Lipton, R. A. DeMillo, and F.G. Sayward. Mutation analysis. Department Comput. Sci. Res. Rep. 155, Yale University, New Haven, Connecticut, April 1979.]]

[4]

J. Chang. Automating specification-based test coverage measurement and test selection. PhD thesis, University of California, Irvine, March 1999.]]

Digital Library

[5]

J. Chang, D. J. Richardson. ADLscope: an automated specification-based unit testing tool. ICS Technical Report 98-26, Department of Information and Computer Science, University of California, Irvine, California, 1998.]]

[6]

J. Chang, D. J. Richardson, and S. Sankar. Structural specification-based testing with ADL. In Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA'96), pp. 62-70, San Diego, California, January 1996.]]

Digital Library

[7]

R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: help for the practicing programmer. Computer, 4( 1 l), April 1978.]]

[8]

R. K. Doong and P. G. Frankl. The ASTOOT approach to testing object-oriented programs. ACM Transactions on Software Engineering and Methodology, April 1994, pp. 101-130.]]

Digital Library

[9]

K. A. Foster. Error sensitive test case analysis (ESTCA). IEEE Transactions on Software Engineering, 6(3):258-264, May 1980.]]

Digital Library

[10]

K. A. Foster. Sensitive test data for logic expressions. ACM SIGSOFTSofmare Engineering Notes, vol. 9, no. 2, pp. 120-126, April 1984.]]

Digital Library

[11]

J. B. Goodenough and S. L. Gerhart. Toward a theory of test data selection. IEEE Transactions on Software Engineering, SE-l(2): 156-173, June 1975.]]

[12]

James Gosling, Bill Joy, and Guy Steele. The Java Language Specification, Addison Wesley, 1996.]]

Digital Library

[13]

R. G. Hamlet. Testing programs with the aid of a compiler. IEEE Transactions on Soshyare Engineering, SE-3(4):279-290, July 1977.]]

Digital Library

[14]

W. E. Howden. Weak mutation testing and completeness of test sets. IEEE Transactions on Sof?ware Engineering, SE-8(4):371-379, July 1982.]]

[15]

M. Hughes and D. Stotts. Daistish: systematic algebraic testing for 00 programs in the presence of side-effects. In Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA), pp.53-61, San Diego, California, January 1996.]]

Digital Library

[16]

P. Jalote. Specification and testing of abstract data types. Comput. Lang, Vol. 17, pp. 75- 82, 1992.]]

Digital Library

[17]

David C. Luckham and Friedrich W. von Henke. An overview of ANNA, a specification language for Ada. IEEE Software, 2(2):9-22, March 1985.]]

Digital Library

[18]

Brian Marick. Generic Coverage Tool (GCT) User's Guide, Testing Foundations, Champaign, Illinois, 1992.]]

[19]

G. J. Myers. The art of software testing. New York: John Wiley and Sons, 1979.]]

Digital Library

[20]

A. Jefferson Offutt. The coupling effect: fact or fiction. In Proceedings of the Third Workshop on Software Testing, Verification, and Analysis, December 1989.]]

Digital Library

[21]

T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 3 1(6):676-686, June 1988.]]

Digital Library

[22]

Robert M. Poston. Automating Specification-based Software Testing, IEEE, 1996.]]

Digital Library

[23]

D. J. Richardson, S. L. Aha, and T. 0. O'Malley. Specification-based test oracles for reactive systems. In Proceedings of the Fourteenth International Conference on Software Engineering, pages 105-l 18, Melbourne, Australia, May 1992.]]

Digital Library

[24]

D. J. Richardson and L. A. Clarke. Partition analysis: a method combining testing and verification. IEEE Transactions on Software Engineering, SE- ll(12): 1477- 1490, December 1985.]]

[25]

S. Sankar and R. Hayes. Specifying and testing software components using ADL. Technical Report SMLI TR-94-23, Sun Microsystems Laboratories, Inc., Mountain View, California, April 1994.]]

Digital Library

[26]

J. M. Spivey. The Z Notation: a Reference Manual. Prentice Hall, New York, 1989.]]

Digital Library

[27]

P. Stocks and D. Carrington. Test template framework: a specification-based testing case study. In Proceedings of the I993 International Symposium on Sofhvare Testing and Analysis (ZSSTA'93), pp. 1 l- 18, Cambridge, Massachusetts, June 1993.]]

Digital Library

[28]

Sun Microsystems Inc., ADL Language Reference Manual, Release 1.1, Mountain View, California, December 1996.]]

[29]

Sun Microsystems Inc. ADLT FIP Download Site. ftp://ftp.uu.net/vendor/adl.]]

[30]

K. C. Tai. Condition-based software testing strategies. In Proceedings ofthe 14th Annual International Computer Software and Applications Conference, 1990.]]

[31]

E. Weyuker, T. Goradia, and A. Singh. Automatically generating test data from a boolean specification. IEEE Transactions on Software Engineering, SE-20(5):353-363, May 1994.]]

Digital Library

[32]

E. J. Weyuker and T. J. Ostrand. Theories of program testing and the application of revealing subdomains. IEEE Transactions on Software Engineering, SE-6(3):236-246, May 1980.]]

Digital Library

[33]

C. J. Van Wyk. Data structures and Cprograms. Addison-Wesley, 1988.]]

Cited By

Yoshida HTokumoto SPrasad MGhosh IUehara TZeller ARoychoudhury A(2016)FSX: fine-grained incremental unit test generation for C/C++ programsProceedings of the 25th International Symposium on Software Testing and Analysis10.1145/2931037.2931055(106-117)Online publication date: 18-Jul-2016
https://dl.acm.org/doi/10.1145/2931037.2931055
Polikarpova NFuria CPei YWei YMeyer BNotkin DCheng BPohl K(2013)What good are strong specifications?Proceedings of the 2013 International Conference on Software Engineering10.5555/2486788.2486823(262-271)Online publication date: 18-May-2013
https://dl.acm.org/doi/10.5555/2486788.2486823
Hatcliff JLeavens GLeino KMüller PParkinson M(2012)Behavioral interface specification languagesACM Computing Surveys10.1145/2187671.218767844:3(1-58)Online publication date: 14-Jun-2012
https://dl.acm.org/doi/10.1145/2187671.2187678
Show More Cited By

Index Terms

Structural specification-based testing: automated support and experimental evaluation
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Software and its engineering
  1. Software notations and tools
    1. Compilers

Recommendations

Structural specification-based testing: automated support and experimental evaluation

In this paper, we describe a testing technique, called structural specification-based testing (SST), which utilizes the formal specification of a program unit as the basis for test selection and test coverage measurement. We also describe an automated ...
Structural specification-based testing with ADL

This paper describes a specification-based black-box technique for testing program units. The main contribution is the method that we have developed to derive test conditions, which are descriptions of test cases, from the formal specification of each ...
Fault-based test suite prioritization for specification-based testing

Context: Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE-7: Proceedings of the 7th European software engineering conference held jointly with the 7th ACM SIGSOFT international symposium on Foundations of software engineering

October 1999

529 pages

ISBN:3540665382

Chairmen:
Oscar Nierstrasz
Univ. Bern, Bern, Switzerland
,
Michel Lemoine
ONERA Centre de Toulouse, Toulouse, France

ACM SIGSOFT Software Engineering Notes Volume 24, Issue 6
Nov. 1999
527 pages
ISSN:0163-5948
DOI:10.1145/318774
Editors:
Oscar Nierstrasz
Univ. Bern, Bern, Switzerland
,
Michel Lemoine
ONERA Centre de Toulouse, Toulouse, France
Issue’s Table of Contents

Sponsors

CEPIS: Council of European Professional Informatics Societies
SIGSOFT: ACM Special Interest Group on Software Engineering
ONERA CERT

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 October 1999

Check for updates

Qualifiers

Article

Conference

SOFT'99/ESEC/FSE'99

Sponsor:

CEPIS
SIGSOFT
ONERA CERT

SOFT'99/ESEC/FSE'99: 7th SIGSOFT Symposium on Foundations of Software Engineering together with 7th European Software Engineering Conference

September 6 - 10, 1999

Toulouse, France

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
914
Total Downloads

Downloads (Last 12 months)119
Downloads (Last 6 weeks)28

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yoshida HTokumoto SPrasad MGhosh IUehara TZeller ARoychoudhury A(2016)FSX: fine-grained incremental unit test generation for C/C++ programsProceedings of the 25th International Symposium on Software Testing and Analysis10.1145/2931037.2931055(106-117)Online publication date: 18-Jul-2016
https://dl.acm.org/doi/10.1145/2931037.2931055
Polikarpova NFuria CPei YWei YMeyer BNotkin DCheng BPohl K(2013)What good are strong specifications?Proceedings of the 2013 International Conference on Software Engineering10.5555/2486788.2486823(262-271)Online publication date: 18-May-2013
https://dl.acm.org/doi/10.5555/2486788.2486823
Hatcliff JLeavens GLeino KMüller PParkinson M(2012)Behavioral interface specification languagesACM Computing Surveys10.1145/2187671.218767844:3(1-58)Online publication date: 14-Jun-2012
https://dl.acm.org/doi/10.1145/2187671.2187678
Batory DSmaragdakis YSiek J(2008)Using modern mathematics as an FOSD modeling languageProceedings of the 7th international conference on Generative programming and component engineering10.1145/1449913.1449921(35-44)Online publication date: 19-Oct-2008
https://dl.acm.org/doi/10.1145/1449913.1449921
Batory D(2008)A Modeling Language for Program Design and SynthesisAdvances in Software Engineering10.1007/978-3-540-89762-0_2(39-58)Online publication date: 27-Nov-2008
https://dl.acm.org/doi/10.1007/978-3-540-89762-0_2
Gargantini A(2007)Using model checking to generate fault detecting testsProceedings of the 1st international conference on Tests and proofs10.5555/1776119.1776130(189-206)Online publication date: 12-Feb-2007
https://dl.acm.org/doi/10.5555/1776119.1776130
Dias Neto ASubramanyan RVieira MTravassos GKraemer EMaletic J(2007)A survey on model-based testing approachesProceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 200710.1145/1353673.1353681(31-36)Online publication date: 5-Nov-2007
https://dl.acm.org/doi/10.1145/1353673.1353681
Rutherford MCarzaniga AWolf AYoung MDevanbu P(2006)Simulation-based test adequacy criteria for distributed systemsProceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering10.1145/1181775.1181804(231-241)Online publication date: 5-Nov-2006
https://dl.acm.org/doi/10.1145/1181775.1181804
Darga PBoyapati C(2006)Efficient software model checking of data structure propertiesACM SIGPLAN Notices10.1145/1167515.116750441:10(363-382)Online publication date: 16-Oct-2006
https://dl.acm.org/doi/10.1145/1167515.1167504
Darga PBoyapati CTarr PCook WBiddle RGabriel R(2006)Efficient software model checking of data structure propertiesProceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications10.1145/1167473.1167504(363-382)Online publication date: 23-Oct-2006
https://dl.acm.org/doi/10.1145/1167473.1167504
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents