How to measure software reliability, and how not to

This paper examines critically, with a view to stimulating a discussion, some concepts which have been used in early work on software reliability measurement, and suggests improvements and areas of potentially fruitful future research. It is proposed that hardware-motivated measures such as mttf, mtbf should not be used for software without justification, and it is shown that such justification may be lacking under quite unexceptionable circumstances. Alternative methods of measuring software reliability are proposed. Emphasis is placed upon differentiating between two concepts of software reliability which are often blurred in the work of previous authors. These are, on the one hand, the reliability of the program-as-it-is (the number of bugs it contains), on the other, the reliability of the program-as-it-performs (failure rate, distribution of time to next failure, etc.). It is argued that the latter, here called operational reliability, is the one we should use. Measures of operational reliability which avoid use of mttf, etc., are proposed. A case is made for software engineers adopting a Bayesian stand-point:both in the interpretation of probability statements and in inference procedures. It is suggested that reliability modelling solely in terms of failures (or number of bugs) is unnecessarily naive. Interest really centres upon the consequences of failures as much as on their frequency. It is proposed that more effort be devoted to the development of models which incorporate a cost (or utility) structure. Finally, brief consideration is given to the question of program structure. The enormous success of hardware reliability theory, in combining component reliabilities with knowledge of system sturcture, must be emulated for software. Unfortunately, software structure does not easily lend itself to such an exercise. Some existing models are considered.