Quantifying the cost of fixing bugs

Quantifying the cost of fixing vs. preventing bugs

When you think about improving software quality, your initial thoughts might be the cost of investing in new tools and engineering labor to implement them. You may also think about adding new processes to the software development cycle to improve quality. However, did you ever think of first focusing on what the cost to your company is to find and fix bugs throughout the product life cycle? Beyond the above direct costs, have you considered the opportunity cost associated with missed release dates or the negative impact on your brand resulting from unhappy customers or product recalls?

Quantify the cost of fixing bugs

In the book “How Google Tests Software – Help me test like Google,” the authors talk about the evolution of Google’s software testing process. The issue of bugs from incomplete testing was one of the biggest roadblocks to Google’s success. Google was able to quantify the cost of poor software quality in terms of the number of engineers involved in fixing bugs. As a result, they put a software quality improvement plan in place that included testing earlier in the software development life cycle and doing continuous testing. This freed up the same engineers to design new features and beat competitors to market. Below is an example of the calculation for a mid-sized project, on an annual basis, using Google’s numbers.

Source Lines of Code (KSLOC) Generated Per Year 200
Average Bugs Per 1000 SLOCx8
Number of Bugs in Code=1600
Average Cost to Fix a Bugx$1,500
Total Yearly Cost of Bug Fixing=$2,400,000
Year Cost of an Engineer/$150,000
Number of Engineers Consumed with Bug Fixing=16
Engineering Team Size/40
Percentage of Staff Used for Bug Fixing 40%
Table 1: Metrics from Google

Use the table above as template to calculate with your own numbers to estimate costs.

It’s likely that when you go through this exercise, you will find that your team is spending 30 to 50% of their time fixing bugs, and that’s just the direct cost. Imagine if you could use more of that time to create new features for your product? Furthermore, what if you can reduce your support costs because fewer bugs are getting to customers? Think about happier customers who purchase more of your products!

Improve your test process maturity to prevent bugs

Companies like Google, who successfully use a continuous software quality approach, have added automated testing to their development cycle, and it pays huge dividends. A key component to Google’s approach is a test platform to ensure that bugs are found as early in the software development life cycle as possible – when bugs cost less to fix.

The table below contains sample costs from Google of bugs found in different phases of testing, and it clearly shows that finding bugs earlier in the development process translates to a lower cost per bug.

Software Testing Phase Where Bugs Were FoundEstimated Cost per Bug
System Testing$5,000
Integration Testing$500
Full Build$50
Unit Testing/Test-Driven Development$5
Table 2: The cost to fix bugs at Google

Now that you have a better idea of what bug fixing costs your team, what is the best way to reduce bugs and improve quality? Well, the average cost to find and fix a bug is not going to go down over time. However, improving and automating your testing process will reduce your overall maintenance cost as shown in the figure below.

Figure 1: The cost of software maintenance drops as the software testing process matures

Companies find themselves at different levels of the spectrum. For example, some may have no formalized testing infrastructure while others, such as Google, have already solved this problem. Your company may find itself somewhere between these two extremes. Where should you start to improve software quality? As with most engineering tasks, a measured, step by step process is the most effective way to improve. Each step should build upon previous improvements as shown in the following figure.

Adopting a software testing process can be easily achieved step by step
Figure 2: Adopting a software testing process can be easily achieved step by step

Conclusion

Companies that adopt and embrace the value of bug prevention, instead of bug fixing, free up valuable engineering resources for product innovation and new product development. As a result, this enables them to gain a greater market share in the industries they serve.

A recommended starting point is to introduce code coverage and static analysis tools. These tools can be implemented without changing the workflow of your development team but will yield actionable metrics to help improve software quality. For example, System Test Automation enables organizations to generate code coverage metrics during any type of software testing and reports on which portions of your source code have been tested. In addition, it reports on which portions of your source code have not been tested.

Further readings and references

Share:

Share on linkedin
Share on twitter
Share on whatsapp

1 thought on “Quantifying the cost of fixing vs. preventing bugs”

Leave a Comment

Related Posts

Improving test efficiency

Improving test efficiency

Effective software teams are always looking for ways to make their members more efficient. They realize that improving the development process is an good way

Recipe for a successful software demo

Recipe for a successful software demo

Software demos are an essential part of a balanced and healthy product development cycle. From internally unveiling a new proof-of-concept to collecting feedback from external

Hey there!

Subscribe and get an email every time we’ve got a new quality piece on here.