[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/1985793.1985801acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

The impact of fault models on software robustness evaluations

Published: 21 May 2011 Publication History

Abstract

Following the design and in-lab testing of software, the evaluation of its resilience to actual operational perturbations in the field is a key validation need. Software-implemented fault injection (SWIFI) is a widely used approach for evaluating the robustness of software components. Recent research [24, 18] indicates that the selection of the applied fault model has considerable influence on the results of SWIFI-based evaluations, thereby raising the question how to select appropriate fault models (i.e. that provide justified robustness evidence). This paper proposes several metrics for comparatively evaluating fault models's abilities to reveal robustness vulnerabilities. It demonstrates their application in the context of OS device drivers by investigating the influence (and relative utility) of four commonly used fault models, i.e. bit flips (in function parameters and in binaries), data type dependent parameter corruptions, and parameter fuzzing. We assess the efficiency of these models at detecting robustness vulnerabilities during the SWIFI evaluation of a real embedded operating system kernel and discuss application guidelines for our metrics alongside.

References

[1]
A. Albinet, J. Arlat, and J. C. Fabre. Characterization of the Impact of Faulty Drivers on the Robustness of the Linux Kernel. In Proc. DSN, pages 867--876, 2004.
[2]
J. Arlat, J. C. Fabre, and M. Rodriguez. Dependability of COTS Microkernel-based Systems. IEEE Trans. Comput., 51(2):138--163, 2002.
[3]
A. Avizienis, J. Laprie, B. Randell, and C. Landwehr. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Trans. Dependable Secure Comput., 1(1):11--33, 2004.
[4]
Boehm, Abts, Clark, Horowitz, Brown, Reifer, Chulani, Madachy, and Steece. Software Cost Estimation with Cocomo II with CD-ROM. Prentice Hall PTR, 2000.
[5]
B. W. Boehm. Software Engineering Economics. Prentice Hall PTR, 1981.
[6]
J. Carreira, H. Madeira, and J. G. Silva. Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers. IEEE Trans. Softw. Eng., 24(2):125--136, 1998.
[7]
A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler. An Empirical Study of Operating Systems Errors. In Proc. SOSP, pages 73--88. ACM, 2001.
[8]
J. Duraes and H. Madeira. Characterization of Operating Systems Behavior in the Presence of Faulty Drivers through Software Fault Emulation. In Proc. PRDC, pages 201--209, 2002.
[9]
A. Ganapathi, V. Ganapathi, and D. Patterson. Windows XP Kernel Crash Analysis. In Proc. LISA, pages 12--22, 2006.
[10]
H. Hecht. Rare Conditions - An Important Cause of Failures. In Proc. COMPASS, pages 81--85, 1993.
[11]
J. C. Huang. An Approach to Program Testing. ACM Comput. Surv., 7(3):113--128, 1975.
[12]
IEEE. Standard Glossary of Software Engineering Terminology. IEEE Std 610.12--1990, page 1, 1990.
[13]
ISO/IEC 19761:2003. Software Engineering - COSMIC-FFP - A Functional Size Measurement Method. 2003.
[14]
ISO/IEC 20926:2003. Software Engineering - IFPUG 4.1 Unadjusted Functional Size Measurement Method - Counting Practices Manual. 2003.
[15]
A. Johansson. Robustness Evaluation of Operating Systems. PhD thesis, TU Darmstadt, 2008.
[16]
A. Johansson and N. Suri. Error Propagation Profiling of Operating Systems. In Proc. DSN, pages 86--95, 2005.
[17]
A. Johansson, N. Suri, and B. Murphy. On the Impact of Injection Triggers for OS Robustness Evaluation. In N. Suri, editor, Proc. ISSRE, pages 127--126, 2007.
[18]
A. Johansson, N. Suri, and B. Murphy. On the Selection of Error Model(s) for OS Robustness Evaluation. In Proc. DSN, pages 502--511, 2007.
[19]
A. Kalakech, K. Kanoun, Y. Crouzet, and J. Arlat. Benchmarking the Dependability of Windows NT4, 2000 and XP. In Proc. DSN, pages 681--686, 2004.
[20]
P. Koopman, J. Sung, C. Dingman, D. Siewiorek, and T. Marz. Comparing Operating Systems using Robustness Benchmarks. In Proc. SRDS, pages 72--79, 1997.
[21]
R. R. Lutz and I. C. Mikulski. Operational Anomalies as a Cause of Safety-Critical Requirements Evolution. Journal of Systems and Software, 65(2):155--161, 2003.
[22]
T. McCabe. A Complexity Measure. IEEE Trans. Softw. Eng., SE-2(4):308--320, 1976.
[23]
M. Mendonça and N. Neves. Robustness Testing of the Windows DDK. In Proc. DSN, pages 554--564, 2007.
[24]
R. Moraes, R. Barbosa, J. Duraes, N. Mendes, E. Martins, and H. Madeira. Injection of Faults at Component Interfaces and Inside the Component Code: Are They Equivalent? In R. Barbosa, editor, Proc. EDCC '06, pages 53--64, 2006.
[25]
MSDN. coredll Module. http://msdn.microsoft.com/en-us/library/aa448387.aspx.
[26]
MSDN. Implementing CEDDK.dll. http://msdn.microsoft.com/en-us/library/ms898217.aspx.
[27]
MSDN. Network Driver Functions. http://msdn.microsoft.com/en-us/library/ms895631.aspx.
[28]
D. Simpson. Windows XP Embedded with Service Pack 1 Reliability, January 2003.
[29]
C. Software. SourceMonitor Version 2.5. http://www.campwoodsw.com/sourcemonitor.html.
[30]
C. Szyperski. Component Software - Beyond Object-Oriented Programming. Addison-Wesley, 1998.
[31]
E. Voas, F. Charron, G. McGraw, K. Miller, and M. Friedman. Predicting how badly "good" Software can behave. IEEE Softw., 14(4):73--83, 1997.
[32]
D. A. Wheeler. SLOCCount. http://www.dwheeler.com/sloccount/.

Cited By

View all
  • (2024)Neural Fault Injection: Generating Software Faults from Natural Language2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S)10.1109/DSN-S60304.2024.00016(23-27)Online publication date: 24-Jun-2024
  • (2022)An Improved Test Case Generation Method based on Test Requirements for Testing Software Component2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C57518.2022.00039(209-218)Online publication date: Dec-2022
  • (2022)Assessment of Security Defense of Native Programs Against Software FaultsSystem Dependability and Analytics10.1007/978-3-031-02063-6_5(69-98)Online publication date: 26-Jul-2022
  • Show More Cited By

Index Terms

  1. The impact of fault models on software robustness evaluations

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICSE '11: Proceedings of the 33rd International Conference on Software Engineering
    May 2011
    1258 pages
    ISBN:9781450304450
    DOI:10.1145/1985793
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 May 2011

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. fault injection
    2. fault models
    3. robustness testing

    Qualifiers

    • Research-article

    Conference

    ICSE11
    Sponsor:
    ICSE11: International Conference on Software Engineering
    May 21 - 28, 2011
    HI, Waikiki, Honolulu, USA

    Acceptance Rates

    Overall Acceptance Rate 276 of 1,856 submissions, 15%

    Upcoming Conference

    ICSE 2025

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)23
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Neural Fault Injection: Generating Software Faults from Natural Language2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S)10.1109/DSN-S60304.2024.00016(23-27)Online publication date: 24-Jun-2024
    • (2022)An Improved Test Case Generation Method based on Test Requirements for Testing Software Component2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C57518.2022.00039(209-218)Online publication date: Dec-2022
    • (2022)Assessment of Security Defense of Native Programs Against Software FaultsSystem Dependability and Analytics10.1007/978-3-031-02063-6_5(69-98)Online publication date: 26-Jul-2022
    • (2021)A Systematic Review on Software Robustness AssessmentACM Computing Surveys10.1145/344897754:4(1-65)Online publication date: 3-May-2021
    • (2021)The Art, Science, and Engineering of Fuzzing: A SurveyIEEE Transactions on Software Engineering10.1109/TSE.2019.294656347:11(2312-2331)Online publication date: 1-Nov-2021
    • (2020)An Investigation on Quality Perspective of Software Functional ArtifactsCrowdsourcing and Probabilistic Decision-Making in Software Engineering10.4018/978-1-5225-9659-2.ch007(109-133)Online publication date: 2020
    • (2020)Towards Generating Realistic and High Coverage Test Data for Constraint-Based Fault InjectionInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402050018730:03(451-479)Online publication date: 28-Apr-2020
    • (2020)ProFIPy: Programmable Software Fault Injection as-a-Service2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)10.1109/DSN48063.2020.00052(364-372)Online publication date: Jun-2020
    • (2019)Threat Modeling the Cloud: An Ontology Based Approach10.1007/978-3-030-12085-6_6(61-72)Online publication date: 30-Jan-2019
    • (2017)Chizpurfle: A Gray-Box Android Fuzzer for Vendor Service Customizations2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2017.16(1-11)Online publication date: Oct-2017
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media