More Web Proxy on the site http://driver.im/

research-article

Emulation of Software Faults: A Field Data Study and a Practical Approach

Authors:

Joao A. Duraes,

Henrique S. MadeiraAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 32, Issue 11

Pages 849 - 867

https://doi.org/10.1109/TSE.2006.113

Published: 01 November 2006 Publication History

Abstract

The injection of faults has been widely used to evaluate fault tolerance mechanisms and to assess the impact of faults in computer systems. However, the injection of software faults is not as well understood as other classes of faults (e.g., hardware faults). In this paper, we analyze how software faults can be injected (emulated) in a source-code independent manner. We specifically address important emulation requirements such as fault representativeness and emulation accuracy. We start with the analysis of an extensive collection of real software faults. We observed that a large percentage of faults falls into well-defined classes and can be characterized in a very precise way, allowing accurate emulation of software faults through a small set of emulation operators. A new software fault injection technique (G-SWFIT) based on emulation operators derived from the field study is proposed. This technique consists of finding key programming structures at the machine code-level where high-level software faults can be emulated. The fault-emulation accuracy of this technique is shown. This work also includes a study on the key aspects that may impact the technique accuracy. The portability of the technique is also discussed and it is shown that a high degree of portability can be achieved.

References

[1]

M. Kalyanakrishnam, Z. Kalbarczyk, and R. Iyer, “Failure Data Analysis of a LAN of Windows NT Based Computers,” Proc. Symp. Reliable Distributed Database Systems (SRDS-18), pp. 178-187, 1999.

Digital Library

[2]

IEEE Trans. Software Eng., vol. 21, no. 5, pp.455-467, May 1995.

Digital Library

[3]

M. Sullivan and R. Chillarege, “Comparison of Software Defects in Database Management Systems and Operating Systems,” Proc. 22nd IEEE Fault Tolerant Computing Symp. (FTCS 22), pp. 475-484, July 1992.

[4]

IEEE Trans. Reliability, vol. 39, no. 4, pp. 409-418, Oct. 1990.

[5]

J. Musa, Software Reliability Engineering. McGraw-Hill, 1996.

[6]

M.R. Lyu, Handbook of Software Reliability Engineering. IEEE Computer Society Press & McGraw-Hill, 1996.

Digital Library

[7]

J. Christmansson and P. Santhanam, “Error Injection Aimed at Fault Removal in Fault Tolerance Mechanisms—Criteria for Error Selection Using Field Data on Software Faults,” Proc. Seventh IEEE Int'l Symp. Software Reliability Eng. (ISSRE '96), Oct./Nov. 1996.

Digital Library

[8]

IEEE Software, vol. 14, no. 4, 1997.

Digital Library

[9]

W.T. Ng and P.M. Chen, “Systematic Improvement of Fault Tolerance in the RIO File Cache,” Proc. IEEE Fault Tolerant Computing Symp. (FTCS), June 1999.

Digital Library

[10]

A. Brown and D. Patterson, “Toward Availability Benchmark: A Case Study of Software RAID Systems,” Proc. USENIX Ann. Technical Conf., pp 263-276, June 2000.

Digital Library

[11]

M. Vieira and H Madeira, “A Dependability Benchmark for OLTP Application Environments,” Proc. 29th Int'l Conf. Very Large Databases (VLDB '03), Sept. 2003.

Digital Library

[12]

J. Zhu, J. Mauro, and I. Pramanick, “R3—A Framework for Availability Benchmarking,” Proc. Int'l Conf. Dependable Systems and Networks (DSN '03), pp. B86-B87, 2003.

[13]

S. Lightstone, J. Hellerstein, W. Tetzlaff, P. Janson, E. Lassettre, C. Norton, B. Rajaraman, and L. Spainhower, “Toward Benchmarking Autonomic Computing Maturity,” Proc. First IEEE Conf. Industrial Automatics (INDIN '03), Aug. 2003.

[14]

K. Kanoun, J. Arlat, D. Costa, M.D. Cin, P. Gil, J.C. Laprie, H. Madeira, and N. Suri, “DBench: Dependability Benchmarking,” Proc. Supplement of the IEEE/IFIP Int'l Conf. Dependable Systems and Networks (DSN '01), 2001.

[15]

J. Christmansson and R. Chillarege, “Generation of an Error Set that Emulates Software Faults,” Proc. 26th IEEE Fault Tolerant Computing Symp. (FTCS 26), pp. 304-313, June 1996.

Digital Library

[16]

J. Durães and H. Madeira, “Characterization of Operating Systems Behavior in the Presence of Faulty Device Drivers through Software Fault Emulation,” Proc. Pacific Rim Int'l Symp. Dependable Computing (PRDC '02), pp. 201-209, 2002.

Digital Library

[17]

J. Durães, M. Vieira, and H. Madeira, “Web Server Dependability Benchmaking,” Proc. Conf. Computer Safety (SAFECOMP), 2004.

[18]

Proc. Int'l Conf. Dependable Systems and Networks (DSN '00), pp. 417-426, 2000.

[19]

R. Chillarege, “Orthogonal Defect Classification,” Handbook of Software Reliability Eng., chapter 9, IEEE CS Press, McGraw-Hill, 1995.

Digital Library

[20]

Proc. ACM SIGSOFT/IEEE Second Workshop Software Testing, Verification, and Analysis, pp. 142-151, July 1988.

[21]

S. Yamada, M. Ohba, and S. Osaki, “S-Shaped Software Reliability Growth Models and Their Application,” IEEE Trans. Reliability, vol. 33, no. 4, pp. 289-292, Oct. 1984.

[22]

IEEE Trans. Software Eng., vol. 19, no. 11, pp. 1055-1070, Nov. 1993.

Digital Library

[23]

T. Khoshgoftaar et al. “Process Measures for Predicting Software Quality,” Proc. High Assurance Systems Eng. Workshop (HASE '97), 1997.

Digital Library

[24]

J.P. Hudepohl et al., “EMERALD: A Case Study in Enhancing Software Reliability,” Proc. IEEE Eighth Int'l Symp. Software Reliability Eng. (ISSRE '98), pp. 85-91, Nov. 1998.

[25]

Proc. 21st IEEE Fault Tolerant Computing Symp. (FTCS 21), pp. 2-9, June 1991.

[26]

R.K. Iyer, “Experimental Evaluation,” Proc. 25th IEEE Symp. Fault Tolerant Computing (FTCS 25), pp. 115-132, June 1995.

Digital Library

[27]

IEEE Trans. Software Eng., vol. 18, no. 11, pp. 943-956, Nov. 1992.

Digital Library

[28]

T. Budd, “Mutation Analysis: Ideas, Examples, Problems, and Prospects,” Computer Program Testing, pp. 129-134, 1981.

[29]

M. Daran and P. Thévenod-Fosse, “Software Error Analysis: A Real Case Study Involving Real Faults and Mutations,” Proc. Third Symp. Software Testing and Analysis (ISSTA '03), pp. 158-171, Jan. 1996.

Digital Library

[30]

IEEE Trans. Computers, vol. 42, no. 8, pp.919-923, Aug. 1993.

Digital Library

[31]

G. Kanawati, N. Kanawati, and J. Abraham, “FERRARI: A Tool for the Validation of System Dependability Properties,” Proc. 22th IEEE Fault Tolerant Computing Symp. (FTCS 22), pp. 336-344, June 1992.

[32]

T. Tsai and R.K. Iyer, “An Approach to Benchmarking of Fault-Tolerant Commercial Systems,” Proc. 26th IEEE Fault Tolerant Computing Symp. (FTCS 26), pp. 314-323, June 1996.

Digital Library

[33]

J. Carreira, H. Madeira, and J.G. Silva, “Xception: Software Fault Injection and Monitoring in Processor Functional Units,” IEEE Trans. Software Eng., vol. 24, no. 2, Feb. 1998.

Digital Library

[34]

J. Aidemark, J. Vinter, P. Folkesson, and J. Karlsson, ”Goofi: Generic Object-Oriented Fault Injection Tool,” Proc. Int'l Conf. Dependable Systems and Networks (DSN '01), July 2001.

Digital Library

[35]

J.-C. Fabre, F. Salles, M. Moreno, and J. Arlat, “Assessment of COTS Microkernels by Fault Injection,” Proc. Seventh IFIP Working Conf. Dependable Computing for Critical Applications (DCCA '99), pp.25-44, 1999.

Digital Library

[36]

P. Koopman, J. Sung, C. Dingman, D. Siewiorek, and T. Marz, “Comparing Operating Systems Using Robustness Benchmarks,” Proc. 16th Int'l Symp. Reliable Distributed Systems (SRDS '97), pp.72-79, 1997.

Digital Library

[37]

IEEE Trans. Reliability, vol. 42, no. 2, pp. 190-204, June 1993.

[38]

W. Kao, “Experimental Study of Software Dependability,” PhD thesis, Technical Report CRHC-94-16, Dept. of Computer Science, Univ. of Illinois at Urbana-Champaign, Illi nois, 1994.

Digital Library

[39]

J. Bieman, D. Dreilinger, and L. Lin, “Using Fault Injection to Increase Test Coverage,” Proc. Seventh IEEE Int'l Symp. Software Reliability Eng. (ISSRE '96), Oct. /Nov. 1996.

Digital Library

[40]

D. Blough and T. Torii, “Fault-Injection-Based Testing of Fault-Tolerant Algorithms in Message-Passing Parallel Computers,” Proc. 27th IEEE Fault Tolerant Computing Symp. (FTCS 27), pp. 258-267, June 1997.

Digital Library

[41]

W.T. Ng, C.M. Aycock, and P.M. Chen, “Comparing Disk and Memory's Resistance to Operating System Crashes,” Proc. Seventh IEEE Int'l Symp. Software Reliability Eng. (ISSRE '96), 1996.

Digital Library

[42]

D. Costa, T. Rilho, and H. Madeira, “Joint Evaluation of Performance and Robustness of a COTS DBMS through Fault-Injection,” Proc. Dependable Systems and Networks Conf. (DSN '00), June 2000.

Digital Library

[43]

D. Costa, T. Rilho, M. Vieira, and H. Madeira, “ESFFI—A Novel Technique for the Emulation of Software Faults in COTS Components,” Proc. Eighth Ann. IEEE Int'l Conf. Eng. of Computer-Based Systems (ECBS '01), Apr. 2001.

[44]

J. Durães and H. Madeira, “Emulation of Software Faults by Selective Mutations at Machine-Code Level,” Proc. 13th IEEE Int'l Symp. Software Reliability Eng. (ISSRE '02), Nov. 2002.

Digital Library

[45]

J. Durães and H. Madeira, “Definition of Software Fault Emulation Operators: A Field Data Study,” Proc. IEEE/IFIP Int'l Conf. Dependable Systems and Networks, Dependable Computing, and Comm. (DSN-DCC '03), June 2003.

[46]

J. Durães and H. Madeira, “Generic Faultloads Based on Software Faults for Dependability Benchmarking,” Proc. IEEE Int'l Symp. Dependable Systems and Networks (DSN '04), 2004.

Digital Library

[47]

Intel Architecture Software Developer's Manual, Vol. 2: Instruction Set Reference, Intel, 1997.

[48]

J. Voas and G. McGraw, Software Fault Injection: Inoculating Programs against Errors. John Wiley and Sons, 1997.

Digital Library

[49]

R.A. DeMillo, R.J. Lipton, and F.G. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.

Digital Library

[50]

R.E. Kirk, Statistics: An Introduction, fourth ed. Assessment Systems Corp., 1999.

Cited By

Campos JCosta EVieira M(2023)Online Failure Prediction for Complex Systems: Methodology and Case StudiesIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2022.319267120:4(3520-3534)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1109/TDSC.2022.3192671
Lavens EPhilippaerts PJoosen W(2022)A Quantitative Assessment of the Detection Performance of Web Vulnerability ScannersProceedings of the 17th International Conference on Availability, Reliability and Security10.1145/3538969.3544416(1-10)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.1145/3538969.3544416
Laranjeiro NAgnelo JBernardino J(2021)A Systematic Review on Software Robustness AssessmentACM Computing Surveys10.1145/344897754:4(1-65)Online publication date: 3-May-2021
https://dl.acm.org/doi/10.1145/3448977
Show More Cited By

Recommendations

On the Emulation of Software Faults by Software Fault Injection
DSN '00: Proceedings of the 2000 International Conference on Dependable Systems and Networks (formerly FTCS-30 and DCCA-8)

This paper presents an experimental study on the emulation of software faults by fault injection. In a first experiment, a set of real software faults has been compared with faults injected by a SWIFI tool (Xception) to evaluate the accuracy of the ...
Emulation of Transient Software Faults for Dependability Assessment: A Case Study
EDCC '10: Proceedings of the 2010 European Dependable Computing Conference

Fault Tolerance Mechanisms (FTMs) are extensively used in software systems to counteract software faults, in particular against faults that manifest transiently, namely Mandelbugs. In this scenario, Software Fault Injection (SFI) plays a key role for ...
Error injection aimed at fault removal in fault tolerance mechanisms-criteria for error selection using field data on software faults
ISSRE '96: Proceedings of the The Seventh International Symposium on Software Reliability Engineering

Fault injection allows a detailed study of complex interactions between faults and fault handling mechanisms. It can be a useful complement to analytical modeling and formal verification techniques in the testing of fault tolerant systems. However, work ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 32, Issue 11

November 2006

79 pages

ISSN:0098-5589

Issue’s Table of Contents

Copyright © 2006.

Publisher

IEEE Press

Publication History

Published: 01 November 2006

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Campos JCosta EVieira M(2023)Online Failure Prediction for Complex Systems: Methodology and Case StudiesIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2022.319267120:4(3520-3534)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1109/TDSC.2022.3192671
Lavens EPhilippaerts PJoosen W(2022)A Quantitative Assessment of the Detection Performance of Web Vulnerability ScannersProceedings of the 17th International Conference on Availability, Reliability and Security10.1145/3538969.3544416(1-10)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.1145/3538969.3544416
Laranjeiro NAgnelo JBernardino J(2021)A Systematic Review on Software Robustness AssessmentACM Computing Surveys10.1145/344897754:4(1-65)Online publication date: 3-May-2021
https://dl.acm.org/doi/10.1145/3448977
Soares JFernandez RSilva MFreitas TMartins R(2021)ZERMIA - A Fault Injector Framework for Testing Byzantine Fault Tolerant ProtocolsNetwork and System Security10.1007/978-3-030-92708-0_3(38-60)Online publication date: 23-Oct-2021
https://dl.acm.org/doi/10.1007/978-3-030-92708-0_3
Cotroneo DDe Simone LLiguori PNatella RBidokhti NDumas MPfahl DApel SRusso A(2019)How bad can a bug get? an empirical analysis of software failures in the OpenStack cloud computing platformProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338916(200-211)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338916
Couceiro RDuarte GDurães JCastelhano JDuarte CTeixeira CBranco Mde Carvalho PMadeira HSarma AMurta L(2019)Biofeedback augmented software engineeringProceedings of the 41st International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER.2019.00018(37-40)Online publication date: 27-May-2019
https://dl.acm.org/doi/10.1109/ICSE-NIER.2019.00018
Honfi DMicskei Z(2019)Classifying generated white-box tests: an exploratory studySoftware Quality Journal10.1007/s11219-019-09446-527:3(1339-1380)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1007/s11219-019-09446-5
Barbosa RCerveira FGonçalo LMadeira H(2019)Emulating representative software vulnerabilities using field dataComputing10.1007/s00607-018-0657-y101:2(119-138)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s00607-018-0657-y
Gabor UDierl SSpinczyk O(2019)Spectrum-Based Fault Localization in Deployed Embedded Systems with Driver Interaction ModelsComputer Safety, Reliability, and Security10.1007/978-3-030-26601-1_7(97-112)Online publication date: 10-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-26601-1_7
Monperrus M(2018)Automatic Software RepairACM Computing Surveys10.1145/310590651:1(1-24)Online publication date: 23-Jan-2018
https://dl.acm.org/doi/10.1145/3105906
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents