What hardware manufacturing can teach software development

We all have experienced the growing impact of software in our lives. We are doing more in software development with each application we build, as the below chart shows. And furthermore, the complexity of what we are trying to do is also growing dramatically. Software is being written to solve increasingly complex real-world problems.

We watch software take control over these complex tasks. But unfortunately, we are also watching the consequences of the failure of that software due to erroneous code. This drives the need to focus more time on rigorous software testing to ensure it is behaving the way it was intended. But how do we ramp up the testing of all this code and still meet our delivery deadlines?

Hardware manufacturing innovation

Looking back 100 years or so to the introduction of assembly lines and manufacturing automation, the goal at that time was to produce as many units in as little time as possible. Quality control was done at the end of the assembly line, once the unit was built. If a unit was defective, it was fixed as a “one-off.”

In the 1950s, Japan embarked on a mission to penetrate Western markets by offering high-quality products. Japanese companies sought out the American business management expert W. Edwards Deming. He advised them to incorporate quality control principles throughout the whole manufacturing process, not just at the end like American companies were doing. Within a few decades, Japanese companies became synonymous with quality. That led to high market penetration due to fierce customer loyalty. It also led to decreased costs, as we will discuss in a minute.

Deming’s approach to quality centered around process improvement. Measure quality at each step in the manufacturing process: design, manufacture, test. When you find a defect, fix the process to remove the defect. Applying this idea to software development means:

• Creating an integrated testing process
  • Designers, developers and testers no longer work in isolation
• Testing continuously
  • No more periodic testing; for instance, only at release time
• Continuous process improvement
  • No more bug fixing – focus on bug prevention

What is a Software Testing Factory?

Leading organizations are applying these principles by establishing a Software Testing Factory. The idea is that each type of testing that an organization performs can be established as an independent assembly line. Some of the assembly lines run very early in the development process, and some assembly lines focus on later, product-centric testing. All assembly lines are able to run continuously and as the software changes and evolves.

The Software Testing Factory is a key approach to establishing and maintaining software quality. It can support:

• Developer-led unit testing (UT) when creating new software, which can find software defects and prevent them from entering the code base
• Software integration testing (IT) to ensure that different software components work correctly when put together
• Black and white box system testing (BBST & WBST) to test at the API level to ensure customer requirements are met, and to perform fault insertion to test error paths
• Continuous metrics generation and reporting of corporate KPIs
• Any kind of repeatable analysis – static analysis, memory profiling, timing analysis

For instance, testing complex code is much easier during early-phase testing, like developer-led unit testing. Individual building blocks can be tested in isolation to ensure they work properly before they are integrated into a wider set of functionalities. Those low-level tests can be re-run any time the code changes to ensure there are no regression errors. On the other hand, inter-process communication might be tested during white box system testing. These tests can also be re-run when the code changes.

In addition to improving software quality, a Software Testing Factory offers these benefits:

• Reduced software development costs
• Faster time-to-market

Let’s take a closer look at the benefits.

Software development costs decrease

• Fixing defects early in the development lifecycle is less expensive than fixing them later. By removing software defects as early in the development process as possible, software development costs decrease. The chart below shows that most defects are introduced during the coding phase. This increases the value of performing developer-led unit testing.
• Engineers will spend much less time finding and fixing bugs. This time is “hidden” as it is lumped into “engineering costs.” Bug fixing takes focus away from developing new features that could capture new customers. In addition, finding and fixing bugs requires a great deal of context switching by engineers. The broken code could have been written weeks or months ago. It takes time for engineers to get back into the mindset they were in when they wrote it.

Faster time-to-market

• Parallel testing provides the ability to “fail faster.”  It shortens the time it takes to identify regression errors which in turn allows for quicker decision-making regarding release readiness.
• The continuous integration provided by a Software Testing Factory means that the product is always ready to release

One key component to look for in a Software Testing Factory is Change Impact Analysis (CIA). CIA helps project teams understand how much retesting needs to happen with each code change. Ideally, the Software Testing Factory should be able to re-execute just the impacted tests as a way to quickly identify regression errors before a full test run is initiated. With the introduction of Agile development, and the increase in the number of tests developed as the project grows, sequential testing of all the tests isn’t feasible for every code change. CIA can be a huge time saver, as the graphic below shows.

Summary

In summary, setting up a Software Testing Factory yields the following benefits for your software development organization:

• Higher quality software due to expanded, repeated testing
• Faster time to market because you are always ready to release; eliminates late-cycle defects
• Reduced cost due to finding and fixing software issues earlier in the lifecycle when they are inexpensive to address

Further resources

The Software Testing Factory Webinar Series