More Web Proxy on the site http://driver.im/

research-article

Comparing the reliability of software systems

Authors:

Vladimir Ivanov,

Giancarlo SucciAuthors Info & Claims

Information Sciences—Informatics and Computer Science, Intelligent Systems, Applications: An International Journal, Volume 423, Issue C

Pages 398 - 411

https://doi.org/10.1016/j.ins.2017.08.079

Published: 01 January 2018 Publication History

Abstract

Assessment of software reliability is inevitable in modern software production process. Many works aimed at better models for measurement and prediction of reliability of software products. Tens of approaches have been developed and evaluated so far. However, very few works focus on approaches to compare existing systems with respect to reliability. Despite a tremendous importance to practice (and software management area), a complete and sound comparison methodology does not exist. In this paper, we propose a methodology for software reliability comparison. The methodology extensively applies the GQM approach and software reliability growth models. The methodology has been thoroughly evaluated on a case of assessment and comparison of three open source mobile operating systems: Sailfish, Tizen and CyanogenMod.

References

[1]

M. Abd-El-Barr, F. Gebali, Reliability analysis and fault tolerance for hypercube multi-computer networks, Inf. Sci. (Ny), 276 (2014) 295-318.

Digital Library

[2]

T. Aldemir, S. Guarro, J. Kirschenbaum, D. Mandelli, L. Mangan, P. Bucci, M. Yau, B. Johnson, C. Elks, E. Ekici, A benchmark implementation of two dynamic methodologies for the reliability modeling of digital instrumentation and control systems, 2009.

[3]

R. Aliev, W. Pedrycz, V. Kreinovich, O. Huseynov, The general theory of decisions, Inf. Sci. (Ny), 327 (2016) 125-148.

Digital Library

[4]

V. Almering, M. van Genuchten, G. Cloudt, P. Sonnemans, Using software reliability growth models in practice, Softw., IEEE, 24 (2007) 82-88.

Digital Library

[5]

C.A. Asad, M.I. Ullah, M.J.U. Rehman, An approach for software reliability model selection, 2004.

[6]

G. Avontuur, K. van der Werff, An implementation of reliability analysis in the conceptual design phase of drive trains, Reliab. Eng. Syst. Saf., 73 (2001) 155-165.

[7]

V.R. Basili, G. Caldiera, H.D. Rombach, The goal question metric approach, Wiley, 1994.

[8]

J. Baussaron, B. Mihaela, G.-R. Lo, G. Fabrice, S. Paul, Reliability assessment based on degradation measurements: how to compare some models?, Reliab. Eng. Syst. Saf., 131 (2014) 236-241.

[9]

M.K. Bhuyan, D.P. Mohapatra, S. Sethi, A survey of computational intelligence approaches for software reliability prediction, SIGSOFT Softw. Eng. Notes, 39 (2014) 1-10.

Digital Library

[10]

K.-Y. Cai, Towards a conceptual framework of software run reliability modeling, Inf. Sci. (Ny), 126 (2000) 137-163.

Digital Library

[11]

K.-Y. Cai, Z. Dong, K. Liu, Software testing processes as a linear dynamic system, Inf. Sci. (Ny), 178 (2008) 1558-1597.

Digital Library

[12]

P. Cao, Z. Dong, K. Liu, K.-Y. Cai, Quantitative effects of software testing on reliability improvement in the presence of imperfect debugging, Inf. Sci. (Ny), 218 (2013) 119-132.

Digital Library

[13]

I.D. Coman, A. Sillitti, G. Succi, Investigating the usefulness of pair-programming in a mature agile team, Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.

[14]

L. Corral, A. Sillitti, G. Succi, A. Garibbo, P. Ramella, Evolution of mobile software development from platform-specific to web-based multiplatform paradigm, ACM, New York, NY, USA, 2011.

[15]

E.O. Costa, G.A. de Souza, A.T.R. Pozo, S.R. Vergilio, Exploring genetic programming and boosting techniques to model software reliability, IEEE Trans. Reliab., 56 (2007) 422-434.

[16]

Z. Ding, M.-H. Chen, X. Li, Online reliability computing of composite services based on program invariants, Inf. Sci. (Ny), 264 (2014) 340-348.

Digital Library

[17]

K.M. El-Said, M.S. El-Sherbeny, Comparing of reliability characteristics between two different systems, Appl. Math. Comput., 173 (2006) 1183-1199.

Digital Library

[18]

N.E. Fenton, M. Neil, A critique of software defect prediction models, IEEE Trans. Softw. Eng., 25 (1999) 675-689.

Digital Library

[19]

N.E. Fenton, N. Ohlsson, Quantitative analysis of faults and failures in a complex software system, IEEE Trans. Softw. Eng., 26 (2000) 797-814.

Digital Library

[20]

N.E. Fenton, S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach, PWS Publishing Co., Boston, MA, USA, 1998.

Digital Library

[21]

O. Fink, E. Zio, U. Weidmann, Quantifying the reliability of fault classifiers, Inf. Sci. (Ny), 266 (2014) 65-74.

Digital Library

[22]

M. Finkelstein, On some comparisons of lifetimes for reliability analysis, Reliab. Eng. Syst. Saf., 119 (2013) 300-304.

[23]

D. Fisch, A. Hofmann, B. Sick, On the versatility of radial basis function neural networks: a case study in the field of intrusion detection, Inf. Sci. (Ny), 180 (2010) 2421-2439.

Digital Library

[24]

S.T. Garren, D.S.P. Richards, General conditions for comparing the reliability functions of systems of components sharing a common environment, J. Appl. Probab., 35 (1998) 124-135.

[25]

S. Gokhale, M. Lyu, K. Trivedi, Software reliability analysis incorporating fault detection and debugging activities, IEEE Computer Society, Washington, DC, USA, 1998.

[26]

M. Gonzlez, J.P. Proenza, L. Almeida, Quantitative characterization of the reliability of simplex buses and stars to compare their benefits in fieldbuses, Reliab. Eng. Syst. Saf., 138 (2015) 163-175.

[27]

S.D. Guikema, A comparison of reliability estimation methods for binary systems, Reliab. Eng. Syst. Saf., 87 (2005) 365-376.

[28]

R.S. Hanmer, D.T. McBride, V.B. Mendiratta, Comparing reliability and security: concepts, requirements, and techniques, Bell Labs Tech. J., 12 (2007) 65-78.

Digital Library

[29]

L. He, X. Zhang, Fuzzy reliability analysis using cellular automata for network systems, Inf. Sci. (Ny), 348 (2016) 322-336.

Digital Library

[30]

L. Hu, F. Sun, H. Xu, H. Liu, X. Zhang, Mutation hopfield neural network and its applications, Inf. Sci. (Ny), 181 (2011) 92-105.

Digital Library

[31]

A. Jatain, Y. Mehta, Metrics and models for software reliability: a systematic review, 2014.

[32]

B. Javadi, D. Kondo, A. Iosup, D. Epema, The failure trace archive: enabling the comparison of failure measurements and models of distributed systems, J. Parallel Distrib. Comput., 73 (2013) 1208-1223.

Digital Library

[33]

A. Jermakovics, A. Sillitti, G. Succi, Mining and visualizing developer networks from version control systems, ACM, 2011.

[34]

J.M. Juran, Managerial Breakthrough, McGraw-Hill, New York, NY, USA, 1964.

[35]

P.K. Kapur, H. Pham, A. Gupta, P.C. Jha, Software Reliability Assessment with OR Applications, Springer London, London, 2011.

[36]

S.-Z. Ke, C.-Y. Huang, K.-L. Peng, Software reliability analysis considering the variation of testing-effort and change-point, ACM, New York, NY, USA, 2014.

[37]

T.M. Khoshgoftaar, T.G. Woodcock, Software reliability model selection: a cast study, IEEE, 1991.

[38]

J. Kirschenbaum, P. Bucci, M. Stovsky, D. Mandelli, T. Aldemir, M. Yau, S. Guarro, E. Ekici, S.A. Arndt, A benchmark system for comparing reliability modeling approaches for digital instrumentation and control systems, Nucl. Technol., 165 (2009) 53-95.

[39]

G.L. Kovcs, S. Drozdik, P. Zuliani, G. Succi, Open source software for the public administration, 2004.

[40]

H. Koziolek, B. Schlich, C. Bilich, A large-scale industrial case study on architecture-based software reliability analysis, 2010.

[41]

U. Kumar, A. Ahmadi, A.K. Verma, P. Varde, Current Trends in Reliability, Availability, Maintainability and Safety: An Industry Perspective, Springer International Publishing, 2015.

[42]

X. Lei, F. Wang, F.-X. Wu, A. Zhang, W. Pedrycz, Protein complex identification through Markov clustering with firefly algorithm on dynamic protein interaction networks, Inf. Sci. (Ny), 329 (2016) 303-316.

Digital Library

[43]

P.L. Li, J. Herbsleb, M. Shaw, Forecasting field defect rates using a combined time-based and metrics-based approach: a case study of openbsd, IEEE, 2005.

[44]

M. Liron, B. Melamed, S.S. Yau, Markov reliability models of fault-tolerant distributed computing systems, Inf. Sci., 40 (1986) 183-206.

Digital Library

[45]

B. Littlewood, Stochastic reliability growth: a model for fault removal in computer programs and hardware design, IEEE Trans. Reliab., 30 (1981) 313-320.

[46]

Lyu, M. R. (Ed.), 1996. Handbook of Software Reliability Engineering. McGraw-Hill, Inc., Hightstown, NJ, USA.

[47]

E.Y. Matsumoto, E. Del-Moral-Hernandez, Improving regression predictions using individual point reliability estimates based on critical error scenarios, Inf. Sci. (Ny), 374 (2016) 65-84.

Digital Library

[48]

F. Maurer, G. Succi, H. Holz, B. Ktting, S. Goldmann, B. Dellen, Software process support over the internet, ACM, 1999.

[49]

I. Moser, M. Gheorghita, A. Aleti, Investigating the correlation between indicators of predictive diagnostic optimisation and search result quality, Inf. Sci. (Ny), 372 (2016) 162-180.

Digital Library

[50]

Q. Mou, Z. Xu, H. Liao, An intuitionistic fuzzy multiplicative best-worst method for multi-criteria group decision making, Inf. Sci. (Ny), 374 (2016) 224-239.

Digital Library

[51]

J. Musa, Handbook of Software Engineering, Van Nostrand Reinhold, 1984.

[52]

J.D. Musa, A. Iannino, K. Okumoto, Software Reliability: Measurement, Prediction, Application, McGraw-Hill, Inc., 1987.

[53]

S.-K. Oh, W. Pedrycz, The design of self-organizing polynomial neural networks, Inf. Sci. (Ny), 141 (2002) 237-258.

Digital Library

[54]

H. Okamura, T. Dohi, A novel framework of software reliability evaluation with software reliability growth models and software metrics, 2014.

[55]

B.-J. Park, S.-K. Oh, W. Pedrycz, The design of polynomial function-based neural network predictors for detection of software defects, Inf. Sci. (Ny), 229 (2013) 40-57.

Digital Library

[56]

W. Pedrycz, Relevancy of fuzzy models, Inf. Sci., 52 (1990) 285-302.

Digital Library

[57]

W. Pedrycz, A. Gacek, Temporal granulation and its application to signal analysis, Inf. Sci. (Ny), 143 (2002) 47-71.

Digital Library

[58]

W. Pedrycz, E. Roventa, A hierarchical neural model of matching, Inf. Sci., 78 (1994) 215-227.

[59]

W. Pedrycz, G. Succi, A. Sillitti, J. Iljazi, Data description: a general framework of information granules, Knowl.-Based Syst., 80 (2015) 98-108.

Digital Library

[60]

H. Pham, Recent Studies in Software Reliability Engineering, Springer, 2003.

[61]

H. Pham, System software reliability, Springer Science & Business Media, 2007.

[62]

C. Rahmani, A.H. Azadmanesh, L. Najjar, A comparative analysis of open source software reliability., JSW, 5 (2010) 1384-1394.

[63]

B. Rossi, B. Russo, G. Succi, Modelling failures occurrences of open source software with reliability growth, in: IFIP Advances in Information and Communication Technology, Vol. 319, Springer Berlin Heidelberg, 2010, pp. 268-280.

[64]

M. Scotto, A. Sillitti, G. Succi, T. Vernazza, A relational approach to software metrics, ACM, 2004.

[65]

R. Sehgal, O. Gandhi, S. Angra, Reliability evaluation and selection of rolling element bearings, Reliab. Eng. Syst. Saf., 68 (2000) 39-52.

[66]

C. Stringfellow, A.A. Andrews, An empirical method for selecting software reliability growth models, Empir. Softw. Eng., 7 (2002) 319-343.

[67]

G. Succi, W. Pedrycz, M. Stefanovic, B. Russo, An investigation on the occurrence of service requests in commercial software applications, Empir. Softw. Engg., 8 (2003) 197-215.

Digital Library

[68]

S.M. Syed-Mohamad, T. McBride, Reliability growth of open source software using defect analysis, IEEE, 2008.

[69]

N. Ullah, A method for predicting open source software residual defects, Softw. Qual. J., 23 (2015) 55-76.

Digital Library

[70]

N. Ullah, M. Morisio, A. Vetro, A comparative analysis of software reliability growth models using defects data of closed and open source software, IEEE, 2012.

[71]

N. Ullah, M. Morisio, A. Vetro, Selecting the best reliability model to predict residual defects in open source software, IEEE Comput., 48 (2015) 50-58.

[72]

A.K. Verma, S. Ajit, D.R. Karanki, Reliability and Safety Engineering, Springer-Verlag London, 2016.

[73]

A. Wood, Predicting software reliability, Computer (Long Beach Calif), 29 (1996) 69-77.

[74]

A. Wood, Software reliability growth models, Tandem Technical Report, 96 (1996).

[75]

A. Yadav, R. Khan, Critical review on software reliability models 1, Int. J. Recent Trends Eng., 2 (2009) 114-116.

[76]

S. Yamada, Software Reliability Modeling Fundamentals and Applications, Springer Japan, 2014.

[77]

Z. Yang, S. Yang, Z. Yu, B. Yin, C. Bai, Graphical User Interface Reliability Prediction Based on Architecture and Event Handler Interaction, Springer International Publishing, Cham, 2015.

[78]

Y. Zhou, J. Davis, Open source software reliability model: an empirical approach, ACM, 2005.

[79]

Q. Zhu, J.-M. Xu, X. Hou, M. Xu, On reliability of the folded hypercubes, Inf. Sci. (Ny), 177 (2007) 1782-1788.

Digital Library

Cited By

Jain KRaghuvanshi KAgarwal ASingh A(2022)A Comprehensive Fault Prediction Model for Improving Software ReliabilityInternational Journal of Software Innovation10.4018/IJSI.29791410:1(1-16)Online publication date: 13-May-2022
https://dl.acm.org/doi/10.4018/IJSI.297914
Gorb AIves ZBonifati AEl Abbadi A(2022)A Recommender Algorithm to Automatically Generate Metrics for GQM Models in Software DevelopmentProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3520244(2539-2541)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3520244
Milovancevic MDimov ASpasov KVračar LPlanić M(2021)Neuro-Fuzzy Evaluation of the Software Reliability Models by Adaptive Neuro Fuzzy Inference SystemJournal of Electronic Testing: Theory and Applications10.1007/s10836-021-05964-y37:4(439-452)Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s10836-021-05964-y
Show More Cited By

Recommendations

Survey of methodologies for quantifying software reliability

An important problem that arises in the testing of software programs is that given piece of source code is reliable or not. Software reliability is an important segment of software quality. So software reliability must be quantified. Quantification refers ...
Toward an effective software reliability evaluation
ICSE '78: Proceedings of the 3rd international conference on Software engineering

Effective software reliability evaluation requires theories of software reliability which define and deal with software reliability quantitatively, technologies for reliability data measurement and data analysis, techniques to estimate or predict ...
A Practical View of Software Measurement and Implementation Experiences Within Motorola
Special issue on software measurement principles, techniques, and environments

A practical view of software measurement that formed the basis for a companywide software metrics initiative within Motorola is described. A multidimensional view of measurement is provided by identifying different dimensions (e.g., metric usefulness/...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Information Sciences: an International Journal

Information Sciences: an International Journal Volume 423, Issue C

January 2018

399 pages

ISSN:0020-0255

Issue’s Table of Contents

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 January 2018

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jain KRaghuvanshi KAgarwal ASingh A(2022)A Comprehensive Fault Prediction Model for Improving Software ReliabilityInternational Journal of Software Innovation10.4018/IJSI.29791410:1(1-16)Online publication date: 13-May-2022
https://dl.acm.org/doi/10.4018/IJSI.297914
Gorb AIves ZBonifati AEl Abbadi A(2022)A Recommender Algorithm to Automatically Generate Metrics for GQM Models in Software DevelopmentProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3520244(2539-2541)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3520244
Milovancevic MDimov ASpasov KVračar LPlanić M(2021)Neuro-Fuzzy Evaluation of the Software Reliability Models by Adaptive Neuro Fuzzy Inference SystemJournal of Electronic Testing: Theory and Applications10.1007/s10836-021-05964-y37:4(439-452)Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s10836-021-05964-y
Wang J(2021)Open source software reliability model with nonlinear fault detection and fault introductionJournal of Software: Evolution and Process10.1002/smr.238533:12Online publication date: 2-Dec-2021
https://dl.acm.org/doi/10.1002/smr.2385
Sudharson DPrabha D(2019)RETRACTED ARTICLE: A novel machine learning approach for software reliability growth modelling with pareto distribution functionSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-019-04047-723:18(8379-8387)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1007/s00500-019-04047-7
Gusmanov K(2019)CNN LSTM Network Architecture for Modeling Software ReliabilitySoftware Technology: Methods and Tools10.1007/978-3-030-29852-4_17(210-217)Online publication date: 15-Oct-2019
https://dl.acm.org/doi/10.1007/978-3-030-29852-4_17
Ivanov VRogers ASucci GYi JZorin VChaudron MCrnkovic IChechik MHarman M(2018)Precooked developer dashboardsProceedings of the 40th International Conference on Software Engineering: Companion Proceeedings10.1145/3183440.3195028(402-403)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3183440.3195028

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents