Enhancing software reliability estimates using modified adaptive testing

Context: Most software reliability models are based on a binary notion of correctness, i.e. "successful" or "failed." However, in several instances, it is important to account of failure severity to obtain more descriptive and accurate estimates of the reliability of the software. Objective: In this paper, we develop a set of extended metrics based on the Nelson's software reliability model to account for information gained from a user's point of view regarding the severity of the observed failures. Model formulation based on multi-granularity failure severity is provided, and the proposed metrics are proved to be backward compatible. Method: In order to estimate the software reliability through testing, an extended adaptive testing strategy, namely Modified Adaptive Testing (MAT) is proposed. The use of test history information allows the resulting test process to be adaptive in the selection of tests under limited test budget. Simulations and experiments on real-life programs are conducted to evaluate the effectiveness of MAT. Results: Data show that the reliability estimates obtained using MAT (a) are closer to the "true" reliability than those obtained using random testing and (b) lead to lower variance than the techniques used for comparison, which means MAT can be applied to help testers and reliability engineers better understand the reliability of their programs. Conclusion: It is concluded that the proposed approach can enhance the software reliability estimation testing by guiding the test case selection process by providing more descriptive and accurate results.