[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
article

A systematic mapping study on the combination of static and dynamic quality assurance techniques

Published: 01 January 2012 Publication History

Abstract

Context: A lot of different quality assurance techniques exist to ensure high quality products. However, most often they are applied in isolation. A systematic combination of different static and dynamic quality assurance techniques promises to exploit synergy effects, such as higher defect detection rates or reduced quality assurance costs. However, a systematic overview of such combinations and reported evidence about achieving synergy effects with such kinds of combinations is missing. Objective: The main goal of this article is the classification and thematic analysis of existing approaches that combine different static and dynamic quality assurance technique, including reported effects, characteristics, and constraints. The result is an overview of existing approaches and a suitable basis for identifying future research directions. Method: A systematic mapping study was performed by two researchers, focusing on four databases with an initial result set of 2498 articles, covering articles published between 1985 and 2010. Results: In total, 51 articles were selected and classified according to multiple criteria. The two main dimensions of a combination are integration (i.e., the output of one quality assurance technique is used for the second one) and compilation (i.e., different quality assurance techniques are applied to ensure a common goal, but in isolation). The combination of static and dynamic analyses is one of the most common approaches and usually conducted in an integrated manner. With respect to the combination of inspection and testing techniques, this is done more often in a compiled way than in an integrated way. Conclusion: The results show an increased interest in this topic in recent years, especially with respect to the integration of static and dynamic analyses. Inspection and testing techniques are currently mostly performed in an isolated manner. The integration of inspection and testing techniques is a promising research direction for the exploitation of additional synergy effects.

References

[1]
T.F. Chang, A. Danylyzsn, S. Norimatsu, J. Rivera, D. Shepard, A. Lattanze, J. Tomayko, "Continuous verification" in mission critical software development, in: Proceedings of the Thirtieth Hawaii International Conference on System Sciences, 1997, pp. 273-284.
[2]
Endres, A. and Rombach, D., A Handbook of Software and Systems Engineering. 2003. Addison Wesley.
[3]
Y. Chen, S. Liu, W.E Wong, A method combining review and testing for verifying software systems, in: Proceedings of the 2008 International Conference on BioMedical Engineering and Informatics, 2008, pp. 827-831.
[4]
A. Hanna, H.Z. Ling, X. Yang, M. Debbabi, A synergy between static and dynamic analysis for the detection of software security vulnerabilities, in: Proceedings of the Confederated International Conferences, CoopIS, DOA, IS, and ODBASE 2009 on On the Move to Meaningful Internet Systems: Part II, 2009, pp. 815-832.
[5]
D.M. Zimmerman, J.R. Kiniry, A verification-centric software development process for Java, in: Proceedings of the 9th International Conference on Quality Software, 2009, pp. 76-85.
[6]
Roper, M., Wood, M. and Miller, J., An empirical evaluation of defect detection techniques. Information and Software Technology. v39 i11. 763-775.
[7]
So, S.S., Cha, S.D., Shimeall, T.J. and Kwon, Y.R., An empirical evaluation of six methods to detect faults in software. Software Testing, Verification and Reliability. v12 i3. 155-171.
[8]
E. Kamsties, C.M. Lott, An empirical evaluation of three defect-detection techniques, in: 5th European Software Engineering Conference, 1995, pp. 362-383.
[9]
R. Conradi, A.S. Marjara, B. Skatevik, An empirical study of inspection and testing data at Ericsson, Norway, in: 24th NASA Software Engineering Workshop, 1999, pp. 1-7.
[10]
C. Andersson, T. Thelin, P. Runeson, N. Dzamashvili, An experimental evaluation of inspection and testing for detection of design faults, in: Proceedings of the 2003 International Symposium on Empirical Software Engineering, 2003, pp. 174-184.
[11]
T. Berling, T. Thelin, An industrial case study of the verification and validation activities, in: Proceedings of the 9th International Symposium on Software Metrics, 2003, pp. 226-238.
[12]
Chen, T.Y., Poon, P.L., Tang, S.F., Tse, T.H. and Yu, Y.T., Applying testing to requirements inspection for software quality assurance. Information Systems Control Journal. v6. 50-56.
[13]
Massicotte, P., Badri, L. and Badri, M., Aspects-classes integration testing strategy: an incremental approach. Rapid Integration of Software Engineering Techniques. v3943. 158-173.
[14]
Godefroid, P., de Halleux, P., Nori, A.V., Rajamani, S.K., Schulte, W. and Tillmann, N., Automating software testing using program analysis. IEEE Software. v25 i5. 30-37.
[15]
C. Csallner, Y. Smaragdakis, Check 'n' crash: combining static checking and testing, in: Proceedings of the 27th International Conference on Software engineering, 2005, pp. 422-431.
[16]
Artho, C. and Biere, A., Combined static and dynamic analysis. Electronic Notes in Theoretical Computer Science. v131. 3-14.
[17]
J. Chen, H. Zhou, S.D. Bruda, Combining model checking and testing for software analysis, in: Proceedings of the 2008 International Conference on Computer Science and Software Engineering, 2008, pp. 206-209.
[18]
Chebaro, O., Kosmatov, N., Giorgetti, A. and Julliand, J., Combining static analysis and test generation for C program debugging. Tests and Proofs. v6143. 94-100.
[19]
P. Centonze, R. Flynn, M. Pistoia, Combining static and dynamic analysis for automatic identification of precise access-control policies, in; Proceedings of the 23rd Annual Computer Security Applications Conference, 2007, pp. 292-303.
[20]
F.D. Anger, R.V. Rodriguez, M. Young, Combining static and dynamic analysis of concurrent programs, in: Proceedings of the International Conference on Software Maintenance, 1994, pp. 89-98.
[21]
M. Wood, M. Roper, A. Brooks, J. Miller, Comparing and combining software defect detection techniques - a replicated empirical study, in: 6th European Software Engineering Conference, 1997, pp. 262-277.
[22]
S. Wagner, J. Jürjens, C. Koller, P. Trischberger, Comparing bug finding tools with reviews and test, in: Proceedings of the 17th International Conference on Testing of Communicating Systems, 2005, pp. 40-55.
[23]
Basili, V.R. and Selby, R.W., Comparing the effectiveness of software testing strategies. IEEE Transactions on Software Engineering. v13 i12. 1278-1296.
[24]
Fagan, M.E., Design and code inspections to reduce errors in program development. IBM Systems Journal. v15 i3. 182-211.
[25]
P. Runeson, A. Andrews, Detection or isolation of defects? An experimental comparison of unit testing and code inspection, in: Proceedings of the 14th International Symposium on Software Reliability Engineering, 2003, pp. 3-13.
[26]
Csallner, C., Smaragdakis, Y. and Xie, T., DSD-Crasher: a hybrid analysis tool for bug finding. ACM Transactions on Software Engineering and Methodology. v17 i2. 1-37.
[27]
S. Zhang, Y. Lin, Z. Gu, J. Zhao, Effective identification of failure-inducing changes: a hybrid approach, in: Proceedings of the 8th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering, 2008, pp. 77-83.
[28]
M. Gopinathan, S.K. Rajamani, Enforcing object protocols by combining static and runtime analysis, in: Proceedings of the 23rd ACM SIGPLAN Conference on Object-oriented Programming Systems Languages and Applications, 2008, pp. 245-260.
[29]
Engineering Village. <http://www.engineeringvillage2.org>.
[30]
Franz, L. and Shih, J., Estimating the value of inspections and early testing for software projects. Hewlett-Packard Journal. v45. 60-67.
[31]
Juristo, N. and Vegas, S., Functional testing, structural testing, and code reading: what fault type do they each detect?. Empirical Methods and Studies in Software Engineering. 208-232.
[32]
B.A. Kitchenham, S. Charters, Guidelines for Performing Systematic Literature Reviews in Software Engineering, Technical Report EBSE-2007-01, Keele University and University of Durham, 2007.
[33]
Q. Chen, L. Wang, Z. Yang, S.D. Stoller, HAVE: detecting atomicity violations via integrated dynamic and static analysis, in: Proceedings of the 12th International Conference on Fundamental Approaches to Software Engineering: Held as Part of the Joint European Conferences on Theory and Practice of Software, 2009, pp. 425-439.
[34]
Health, Social, and Economic Research, The Economic Impacts of Inadequate Infrastructure for Software Testing, National Institute of Standards and Technology, 2002.
[35]
Q. Chen, L. Wang, Z. Yang, HEAT: an integrated static and dynamic approach for thread escape analysis, in: Proceedings of the 33rd Annual IEEE International Computer Software and Applications Conference, 2009, pp. 142-147.
[36]
P.D. Kumar, A. Nema, R. Kumar, Hybrid analysis of executables to detect security vulnerabilities: security vulnerabilities, in: Proceedings of the 2nd India Software Engineering Conference, 2009, pp. 141-148.
[37]
IEEE Standard 610.12-1990. IEEE Standard Glossary of Software Engineering Terminology, 1990.
[38]
D. Winkler, B. Riedl, S. Biffl, Improvement of design specifications with inspection and testing, in: Proceedings of the 31st Euromicro Conference on Software Engineering and Advanced Applications, 2005, pp. 222-231.
[39]
F. Lanubile, T. Mallardo, Inspecting automated test code: a preliminary study, in: Proceedings of the 8th International Conference on Agile Processes in Software Engineering and Extreme Programming, 2007, pp. 115-122.
[40]
N. Ward, Integrated formal verification and validation of safety critical software, in: Proceedings of the Aerospace Software Engineering for Advanced Systems Architectures Conference, 1993, pp. 10-13.
[41]
S. Liu, Integrating specification-based review and testing for detecting errors in programs, in: Proceedings of the 9th International Conference on Formal Methods and Software Engineering, 2007, pp. 136-150.
[42]
A. Aggarwal, P. Jalote, Integrating static and dynamic analysis for detecting vulnerabilities, in: 30th Annual International Conference on Computer Software and Applications, 2006, pp. 343-350.
[43]
G.A.D. Lucca, M.D. Penta, Integrating static and dynamic analysis to improve the comprehension of existing web applications, in: Proceedings of the 7th IEEE International Symposium on Web Site Evolution, 2005, pp. 87-94.
[44]
S. Liu, T. Tamai, S. Nakajima, Integration of formal specification, review, and testing for software component quality assurance, in: Proceedings of the 2009 ACM symposium on Applied Computing, 2009, pp. 415-421.
[45]
Winkler, D., Biffl, S. and Faderl, K., Investigating the temporal behavior of defect detection in software inspection and inspection-based testing. Product-Focused Software Process Improvement. v6156. 17-31.
[46]
M. Klaes, F. Elberzhager, H. Nakao, Managing software quality through a hybrid defect content and effectiveness model, in: Proceedings of the 2nd ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, 2008, pp. 321-323.
[47]
F. Elberzhager, J. Muench, D. Rombach, B. Freimut, Optimizing cost and quality by integrating inspection and test processes, in: International Conference on Software and Systems Process, 2011, pp. 3-12.
[48]
Wiegers, K.E., Peer Reviews in Software. 2002. Addison-Wesley.
[49]
Burnstein, I., Practical Software Testing. 2002. Springer.
[50]
P. Joshi, K. Sen, M. Shlimovich, Predictive testing: amplifying the effectiveness of software testing, in: Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2007, pp. 561-564.
[51]
P. Jalote, V. Vangala, T. Singh, P. Jain, Program partitioning: a framework for combining static and dynamic analysis, in: Proceedings of the 2006 International Workshop on Dynamic Systems Analysis, 2006, pp. 11-16.
[52]
Juristo, N., Moreno, A.M. and Vegas, S., Reviewing 25 years of testing technique experiments. Empirical Software Engineering. v9 i1-2. 7-44.
[53]
D. Balzarotti, M. Cova, V. Felmetsger, N. Jovanovic, E. Kirda, C. Kruegel, G. Vigna, Saner: composing static and dynamic analysis to validate sanitization in web applications, in: Proceedings of the 2008 IEEE Symposium on Security and Privacy, 2008, pp. 387-401.
[54]
P. Strooper, M. Wojcicki, Selecting V&V technology combinations: how to pick a winner? in: Proceedings of the 12th International Conference on Engineering Complex Computer Systems, 2007, pp. 87-96.
[55]
Pressman, R., Software Engineering: A Practitioner's Approach. 2000. fifth ed. McGraw-Hill, London.
[56]
Gilb, T. and Graham, D., Software Inspections. 1993. Addison-Wesley.
[57]
Jones, C., Software project management practices: failure versus success. CrossTalk - The Journal of Defense Software Engineering. v17 i10. 5-9.
[58]
Aurum, A., Petersson, H. and Wohlin, C., State-of-the-art: software inspections after 25 years. Software Testing, Verification and Reliability. v12 i3. 133-154.
[59]
O. Laitenberger, Studying the effects of code inspection and structural testing on software quality, in: Proceedings of the 9th International Symposium on Software Reliability Engineering, 1998, pp. 237-246.
[60]
S. Kollanus, J. Koskinen, Survey of software inspection research: 1991-2005, Computer Science and Information Systems reports, Working Papers WP-40, University of Jyväskylä, Finland, 2007.
[61]
K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, Systematic mapping studies in software engineering, in: Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, 2008, pp. 1-10.
[62]
F. Iturbe, Systematic testing and reviewing, in: Proceedings of the 11th International Conference on Software Engineering and Knowledge Engineering, 1999, pp. 295-299.
[63]
A. Gupta, P. Jalote, Test inspected unit or inspect unit tested code? in: Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement, 2007, pp. 51-60.
[64]
M.J. Harrold, Testing: a roadmap, in: International Conference on Software Engineering, The Future of Software Engineering, 2000, pp. 61-72.
[65]
Landis, J.R. and Koch, G.G., The measurement of observer agreement for categorical data. Biometrics. v33 i1. 159-174.
[66]
Anderson, P., The use and limitations of static-analysis tools to improve software quality. CrossTalk: The Journal of Defense Software Engineering. v21 i6. 18-21.
[67]
A.V. Nori, S.K. Rajamani, S. Tetali, A.V. Thakur, The Yogi project: software property checking via static analysis and testing, in: Proceedings of the 15th International Conference on Tools and Algorithms for the Construction and Analysis of Systems: Held as Part of the Joint European Conferences on Theory and Practice of Software, 2009, pp. 178-181.
[68]
J. Chen, S. MacDonald, Towards a better collaboration of static and dynamic analyses for testing concurrent programs, in: Proceedings of the 6th Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging, 2008, pp. 1-9.
[69]
F. Elberzhager, R. Eschbach, Towards reduction of test effort: predicting defect-prone code classes and expected defect types based on inspection results, in: 36th Euromicro Software Engineering and Advanced Application, Proceedings of the Work in Progress Session, 2010.
[70]
A. Avancini, M. Ceccato, Towards security testing with taint analysis and genetic algorithms, in: Proceedings of the 2010 ICSE Workshop on Software Engineering for Secure Systems, 2010, pp. 65-71.
[71]
F. Elberzhager, R. Eschbach, J. Muench, Using inspection results for prioritizing test activities, in: 21st International Symposium on Software Reliability Engineering, Supplemental Proceedings, 2010, pp. 263-272. <http://inspection.iese.fhg.de/?p=documents>.
[72]
Duke, E., V&V of flight and mission-critical software. IEEE Software. v6 i3. 39-45.
[73]
Runeson, P., Andersson, C., Thelin, T., Andrews, A. and Berling, T., What do we know about defect detection methods?. IEEE Software. v23 i3. 82-90.
[74]
Zotero Tool. <http://www.zotero.org/>.

Cited By

View all
  • (2022)Survey of Approaches for Postprocessing of Static Analysis AlarmsACM Computing Surveys10.1145/349452155:3(1-39)Online publication date: 3-Feb-2022
  • (2022)Advances in database systems education: Methods, tools, curricula, and way forwardEducation and Information Technologies10.1007/s10639-022-11293-028:3(2681-2725)Online publication date: 31-Aug-2022
  • (2019)An Efficient Algorithm for Combining Verification and Validation MethodsSOFSEM 2019: Theory and Practice of Computer Science10.1007/978-3-030-10801-4_26(324-340)Online publication date: 27-Jan-2019
  • Show More Cited By
  1. A systematic mapping study on the combination of static and dynamic quality assurance techniques

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image Information and Software Technology
    Information and Software Technology  Volume 54, Issue 1
    January, 2012
    137 pages

    Publisher

    Butterworth-Heinemann

    United States

    Publication History

    Published: 01 January 2012

    Author Tags

    1. Combination
    2. Dynamic quality assurance
    3. Inspection
    4. Static quality assurance
    5. Systematic mapping study
    6. Testing

    Qualifiers

    • Article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)0
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 21 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)Survey of Approaches for Postprocessing of Static Analysis AlarmsACM Computing Surveys10.1145/349452155:3(1-39)Online publication date: 3-Feb-2022
    • (2022)Advances in database systems education: Methods, tools, curricula, and way forwardEducation and Information Technologies10.1007/s10639-022-11293-028:3(2681-2725)Online publication date: 31-Aug-2022
    • (2019)An Efficient Algorithm for Combining Verification and Validation MethodsSOFSEM 2019: Theory and Practice of Computer Science10.1007/978-3-030-10801-4_26(324-340)Online publication date: 27-Jan-2019
    • (2017)Performance-Based Comparative Assessment of Open Source Web Vulnerability ScannersSecurity and Communication Networks10.1155/2017/61581072017Online publication date: 24-May-2017
    • (2017)A mapping study on design-time quality attributes and metricsJournal of Systems and Software10.1016/j.jss.2017.01.026127:C(52-77)Online publication date: 1-May-2017
    • (2016)The usage of ISBSG data fields in software effort estimationJournal of Systems and Software10.1016/j.jss.2015.11.040113:C(188-215)Online publication date: 1-Mar-2016
    • (2016)A systematic literature review of literature reviews in software testingInformation and Software Technology10.1016/j.infsof.2016.09.00280:C(195-216)Online publication date: 1-Dec-2016
    • (2016)When and what to automate in software testing? A multi-vocal literature reviewInformation and Software Technology10.1016/j.infsof.2016.04.01576:C(92-117)Online publication date: 1-Aug-2016
    • (2015)Perspectives on static analysis of mobile apps (invited talk)Proceedings of the 3rd International Workshop on Software Development Lifecycle for Mobile10.1145/2804345.2804352(29-30)Online publication date: 31-Aug-2015
    • (2015)Cost, benefits and quality of software development documentationJournal of Systems and Software10.1016/j.jss.2014.09.04299:C(175-198)Online publication date: 1-Jan-2015
    • Show More Cited By

    View Options

    View options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media