When you think about improving software quality, your initial thoughts might be the cost of investing in new tools and engineering labor to implement them. You may also think about adding new processes to the software development cycle to improve quality. However, did you ever think of first focusing on what the cost to your company is to find and fix bugs throughout the product life cycle? Beyond the above direct costs, have you considered the opportunity cost associated with missed release dates or the negative impact on your brand resulting from unhappy customers or product recalls?

Quantify the cost of fixing bugs

In the book “How Google Tests Software – Help me test like Google,” the authors talk about the evolution of Google’s software testing process. The issue of bugs from incomplete testing was one of the biggest roadblocks to Google’s success. Google was able to quantify the cost of poor software quality in terms of the number of engineers involved in fixing bugs. As a result, they put a software quality improvement plan in place that included testing earlier in the software development life cycle and doing continuous testing. This freed up the same engineers to design new features and beat competitors to market. Below is an example of the calculation for a mid-sized project, on an annual basis, using Google’s numbers.

Source Lines of Code (KSLOC) Generated Per Year 200
Average Bugs Per 1000 SLOCx8
Number of Bugs in Code=1600
Average Cost to Fix a Bugx$1,500
Total Yearly Cost of Bug Fixing=$2,400,000
Year Cost of an Engineer/$150,000
Number of Engineers Consumed with Bug Fixing=16
Engineering Team Size/40
Percentage of Staff Used for Bug Fixing 40%
Table 1: Metrics from Google

Use the table above as template to calculate with your own numbers to estimate costs.

It’s likely that when you go through this exercise, you will find that your team is spending 30 to 50% of their time fixing bugs, and that’s just the direct cost. Imagine if you could use more of that time to create new features for your product? Furthermore, what if you can reduce your support costs because fewer bugs are getting to customers? Think about happier customers who purchase more of your products!

Improve your test process maturity to prevent bugs

Companies like Google, who successfully use a continuous software quality approach, have added automated testing to their development cycle, and it pays huge dividends. A key component to Google’s approach is a test platform to ensure that bugs are found as early in the software development life cycle as possible – when bugs cost less to fix.

The table below contains sample costs from Google of bugs found in different phases of testing, and it clearly shows that finding bugs earlier in the development process translates to a lower cost per bug.

Software Testing Phase Where Bugs Were FoundEstimated Cost per Bug
System Testing$5,000
Integration Testing$500
Full Build$50
Unit Testing/Test-Driven Development$5
Table 2: The cost to fix bugs at Google

Now that you have a better idea of what bug fixing costs your team, what is the best way to reduce bugs and improve quality? Well, the average cost to find and fix a bug is not going to go down over time. However, improving and automating your testing process will reduce your overall maintenance cost as shown in the figure below.

Figure 1: The cost of software maintenance drops as the software testing process matures

Companies find themselves at different levels of the spectrum. For example, some may have no formalized testing infrastructure while others, such as Google, have already solved this problem. Your company may find itself somewhere between these two extremes. Where should you start to improve software quality? As with most engineering tasks, a measured, step by step process is the most effective way to improve. Each step should build upon previous improvements as shown in the following figure.

Adopting a software testing process can be easily achieved step by step
Figure 2: Adopting a software testing process can be easily achieved step by step

Conclusion

Companies that adopt and embrace the value of bug prevention, instead of bug fixing, free up valuable engineering resources for product innovation and new product development. As a result, this enables them to gain a greater market share in the industries they serve.

A recommended starting point is to introduce code coverage and static analysis tools. These tools can be implemented without changing the workflow of your development team but will yield actionable metrics to help improve software quality. For example, System Test Automation enables organizations to generate code coverage metrics during any type of software testing and reports on which portions of your source code have been tested. In addition, it reports on which portions of your source code have not been tested.

Further readings and references

Share:

Share on linkedin
Share on twitter
Share on whatsapp

3 thoughts on “Cost of fixing vs. preventing bugs”

  1. Testing is important, but you are missing a point.
    You said ” … bug prevention, instead of bug fixing, free up valuable engineering resources for product innovation and new product development”
    That is true, but then you said:
    “A recommended starting point is to introduce code coverage and static analysis tools”
    None of the above help prevent bugs. They can only detect them after they exist.
    Your post is good, but understand that bugs, like any quality factor, are prevented from inception, design, coding practices, people management, leadership…
    Testing is only one part of it, arguably, not the main tool because it can only be executed after the fact.
    Quoting E. Deming:
    “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.”
    https://www.ifm.eng.cam.ac.uk/research/dstools/demings-14-points/

    Reply
    • Alex-

      Very good points indeed. There are definitely situations where quality “after the fact” makes sense. One is that there are large, legacy code bases out there that were not built with quality in mind. And now the owners of that code what it to pass a safety certification. Testing is the only way to get there as the design and coding are already done. And in fact the strategies outlined in the article probably are best used in this situation.

      Also, once you have developed a high quality piece of software, and you want to add to it, it can be harder to maintain that high level of quality, thus incurring a lot of technical debt. So testing can be a critical piece in ensuring that something that used to work is still working after a software change is made. Keeping the architecture in sync with the code is also another good strategy, and there are tools for that as well.

      Thanks for your interest in this topic!

      Lynda

      Reply

Leave a Comment

Related Posts

Improving test efficiency
John Paliotta

Improving test efficiency

Effective software teams are always looking for ways to make their members more efficient. They realize that improving the development process is an good way

Debugging without Security risks
Niroshan Rajadurai

Debugging without security risks

Security researcher Michael Myng found the keylogging code in software drivers that were preinstalled on HP laptops to make the keyboard work. He discovered the

Error: Safety security both
Sebastian Krings

A simple error: Safety, security or both?

Aside from safety properties, can we use static analysis tools to detect security issues? Yes, as we will show by discussing a programming error in uftpd,

Isolate verify merge_Coderskitchen
Niroshan Rajadurai

Isolate, Verify and Merge

The cost of fixing software is well understood. Much research has been done to demonstrate that fixing defects at the time of introducing them is

Hey there!

Subscribe and get an email every time we’ve got a new quality piece on here.