When you think about improving software quality, your initial thoughts might be the cost of investing in new tools and engineering labor to implement them. You may also think about adding new processes to the software development cycle to improve quality. However, did you ever think of first focusing on what the cost to your company is to find and fix bugs throughout the product life cycle? Beyond the above direct costs, have you considered the opportunity cost associated with missed release dates or the negative impact on your brand resulting from unhappy customers or product recalls?
Quantify the cost of fixing bugs
In the book “How Google Tests Software – Help me test like Google,” the authors talk about the evolution of Google’s software testing process. The issue of bugs from incomplete testing was one of the biggest roadblocks to Google’s success. Google was able to quantify the cost of poor software quality in terms of the number of engineers involved in fixing bugs. As a result, they put a software quality improvement plan in place that included testing earlier in the software development life cycle and doing continuous testing. This freed up the same engineers to design new features and beat competitors to market. Below is an example of the calculation for a mid-sized project, on an annual basis, using Google’s numbers.
| Source Lines of Code (KSLOC) Generated Per Year | 200 | |
| Average Bugs Per 1000 SLOC | x | 8 |
| Number of Bugs in Code | = | 1600 |
| Average Cost to Fix a Bug | x | $1,500 |
| Total Yearly Cost of Bug Fixing | = | $2,400,000 |
| Year Cost of an Engineer | / | $150,000 |
| Number of Engineers Consumed with Bug Fixing | = | 16 |
| Engineering Team Size | / | 40 |
| Percentage of Staff Used for Bug Fixing | 40% |
Use the table above as template to calculate with your own numbers to estimate costs.
It’s likely that when you go through this exercise, you will find that your team is spending 30 to 50% of their time fixing bugs, and that’s just the direct cost. Imagine if you could use more of that time to create new features for your product? Furthermore, what if you can reduce your support costs because fewer bugs are getting to customers? Think about happier customers who purchase more of your products!
Improve your test process maturity to prevent bugs
Companies like Google, who successfully use a continuous software quality approach, have added automated testing to their development cycle, and it pays huge dividends. A key component to Google’s approach is a test platform to ensure that bugs are found as early in the software development life cycle as possible – when bugs cost less to fix.
The table below contains sample costs from Google of bugs found in different phases of testing, and it clearly shows that finding bugs earlier in the development process translates to a lower cost per bug.
| Software Testing Phase Where Bugs Were Found | Estimated Cost per Bug |
| System Testing | $5,000 |
| Integration Testing | $500 |
| Full Build | $50 |
| Unit Testing/Test-Driven Development | $5 |
Now that you have a better idea of what bug fixing costs your team, what is the best way to reduce bugs and improve quality? Well, the average cost to find and fix a bug is not going to go down over time. However, improving and automating your testing process will reduce your overall maintenance cost as shown in the figure below.
Companies find themselves at different levels of the spectrum. For example, some may have no formalized testing infrastructure while others, such as Google, have already solved this problem. Your company may find itself somewhere between these two extremes. Where should you start to improve software quality? As with most engineering tasks, a measured, step by step process is the most effective way to improve. Each step should build upon previous improvements as shown in the following figure.
Conclusion
Companies that adopt and embrace the value of bug prevention, instead of bug fixing, free up valuable engineering resources for product innovation and new product development. As a result, this enables them to gain a greater market share in the industries they serve.
A recommended starting point is to introduce code coverage and static analysis tools. These tools can be implemented without changing the workflow of your development team but will yield actionable metrics to help improve software quality. For example, System Test Automation enables organizations to generate code coverage metrics during any type of software testing and reports on which portions of your source code have been tested. In addition, it reports on which portions of your source code have not been tested.
1 thought on “Quantifying the cost of fixing vs. preventing bugs”
Thanks Lynda, I enjoyed reading your article + the handy links at the bottom. Kurt