After an issue in testing, that cost a company more than $440 Million, the question of “what is the cost of a critical defect/bug being carried over to customers?” has become a critical issue. Now, reliability of the software is critical in software applications that interact with patients in hospitals, critical in high impact software applications such as market making software used in stock exchanges, high volume transaction applications such as eCommerce stores, market places and software platforms such as apps stores, that support hundred and thousands of applications.
This problem becomes non-trivial because of the following reasons. The cost of fixing the bugs rises exponentially, in each stage of the software development life cycle. i.e. Bugs caught earlier in the development cycle cost lesser to fix than bug caught by customers, as the over heads of time and resources in the process of the bug being reported and then passed through the support team, then to QA Team and then to software developers adds 10x to cost of bugs being fixed by the development teams.
Now, this becomes interesting, as achieving 100% bug free software is not possible, i.e. perfect bug free applications can never be achieved. So what can a Software developer or a project manager aim for in in the development of the application. First would be to figure out the reliability level required in that particular software application, driven by both the conditions in which the software will be used and the business requirements.
The Trade off between cost of testing and required reliability, becomes apparent in any software development live cycle is the above curve, where it gets more expensive to find the next incremental bug after the bulk of the software bugs have been identified.
The X-axis is Reliability of the Software and the Y-axis is Cost. Here you can see the top most curve, which is the cumulative cost, this first drop and then starts to rise as the cost of testing rises in a continuous manner, while the reliability of the software keeps increasing. i.e. to get perfectly reliable software there is an exponential rise in the cost of testing after a certain point. This is the non-trivial problem in software systems that require whey high reliability (such as 1 defect in 1 million transactions). In normal applications the software developers need to decided on the required reliability of the application and then measure the incremental costs of testing the application to draw the above graph. i.e. it is better to determine, incrementally the cost of testing and reliability than trying to hit a mythical number based on very few data points and then extrapolating to the needed reliability level need in that software product.