Abstract
Code coverage is usually used as a measurement of testing quality and as adequacy criterion. Unfortunately, code coverage is very sensitive to modifications of the code structure, and, therefore, the same test suite can achieve different degrees of coverage on the same program written in two syntactically different ways. For this reason, code coverage can provide the tester with misleading information.
In order to understand how a testing criterion is affected by code structure modifications, we introduce a way to measure the sensitivity of coverage to code changes. We formalize the modifications of the code structure using semantic preserving code-to-code transformations and we propose a framework to evaluate coverage robustness to these transformations, extending actual existing coverage criteria.
This allows us to define which programs and which test suites can be considered robust with respect to a certain set of transformations. We can identify when the obtained coverage is fragile and we extend the concept of coverage criterion by introducing an index that measures the fragility of the coverage of a given test suite. We show how to compute the fragility index and we evidence that also well-written industrial code and realistic test suites can be fragile. Moreover, we suggest how to deal with this kind of testing fragility.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Refactoring catalog - website, http://www.refactoring.com/catalog/
Aho, A.V., Sethi, R., Ullman, J.D.: Compilers: principles, techniques, and tools. Addison-Wesley Longman Publishing Co., Inc., Boston (1986)
Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008)
ARP 4761, Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment. Aerospace Recommended Practice, Society of Automotive Engineers, Detroit, USA (1996)
Chilenski, J.J., Miller, S.P.: Applicability of modified condition/decision coverage to software testing. Software Engineering Journal 9(5), 193–200 (1994)
Dijkstra, E.W.: Notes on structured programming. In: Dahl, O.J., Dijkstra, E.W., Hoare, C.A.R. (eds.) Structured Programming. Academic Press (1972)
Do, H., Elbaum, S.G., Rothermel, G.: Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering 10(4), 405–435 (2005)
Fowler, M.: Refactoring: Improving the Design of Existing Code. Addison-Wesley (August 1999)
Fraser, G., Arcuri, A.: Evosuite: Automatic test suite generation for object-oriented software. In: Proc. of ACM SIGSOFT ESEC/FSE, pp. 416–419 (2011)
Goodenough, J.B., Gerhart, S.L.: Toward a theory of test data selection. IEEE Trans. Softw. Eng. 1(2), 156–173 (1975)
Harman, M., Hu, L., Hierons, R., Wegener, J., Sthamer, H., Baresel, A., Roper, M.: Testability transformation. IEEE Trans. Softw. Eng. 30, 3–16 (2004)
Kirner, R.: Towards preserving model coverage and structural code coverage. EURASIP J. Emb. Sys., 1–16 (2009)
Marick, B., Smith, J., Jones, M.: How to misuse code coverage. In: Proc. of ICTCS 1999 (June 1999)
Miller, J.C., Maloney, C.J.: Systematic mistake analysis of digital computer programs. Commun. ACM 6, 58–63 (1963)
Myers, G.J.: The art of software testing, 2nd edn. Wiley (2004)
Namin, A.S., Andrews, J.H.: The influence of size and coverage on test suite effectiveness. In: Proc. of ISSTA 2009, pp. 57–68. ACM (2009)
Rajan, A., Whalen, M.W., Heimdahl, M.P.: The effect of program and model structure on mc/dc test adequacy coverage. In: Proc. of ICSE, pp. 161–170 (2008)
Rapps, S., Weyuker, E.: Selecting software test data using data flow information. IEEE Trans. Soft. Eng. SE-11(4), 367–375 (1985)
Senko, M.E.: A control system for logical block diagnosis with data loading. Commun. ACM 3, 236–240 (1960)
Staats, M., Gay, G., Whalen, M., Heimdahl, M.: On the Danger of Coverage Directed Test Case Generation. In: de Lara, J., Zisman, A. (eds.) FASE 2012. LNCS, vol. 7212, pp. 409–424. Springer, Heidelberg (2012)
Staats, M., Pǎsǎreanu, C.: Parallel symbolic execution for structural test generation. In: Proc. of ISSTA, pp. 183–194. ACM, New York (2010)
Weißleder, S.: Simulated satisfaction of coverage criteria on uml state machines. In: Proc. of ICST 2010, pp. 117–126. IEEE Computer Society (2010)
Weyuker, E.J.: Translatability and decidability questions for restricted classes of program schemas. SIAM J. Comput. 8(4), 587–598 (1979)
Zhu, H., Hall, P., May, J.: Software unit test coverage and adequacy. ACM Computing Surveys 29(4), 366–427 (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 IFIP International Federation for Information Processing
About this paper
Cite this paper
Gargantini, A., Guarnieri, M., Magri, E. (2012). Extending Coverage Criteria by Evaluating Their Robustness to Code Structure Changes. In: Nielsen, B., Weise, C. (eds) Testing Software and Systems. ICTSS 2012. Lecture Notes in Computer Science, vol 7641. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34691-0_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-34691-0_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34690-3
Online ISBN: 978-3-642-34691-0
eBook Packages: Computer ScienceComputer Science (R0)