In the book How Google Tests Software – Help me test like Google, the authors describe how bugs resulting from incomplete testing are one of the biggest barriers to Google’s continuous improvement and success. After analyzing their testing environment, they found 3 key issues:
- For complete testing, developers must write 2-3 lines of “Test Code” for every line of code being tested
- This “Test Code” requires the same amount of maintenance as the application code and often has more bugs than the application code
- Most developers do not perform complete testing
Google’s test strategy
Google’s solution was to create a well-defined and team-focused approach to testing that supports continuous integration. The “hyper incremental” builds of the application serve as the norm and ensure that the developers are responsible for quality.
Use test automation
You may ask the question “Can my organization adopt this process?” With a massive amount of resources, it may be possible. However, this issue can be an obstacle for many development teams. But you can build something similar by utilizing test automation.
Comparison of testing processes at Google and the a corresponding Test Automation tool-set.
|Google Software Testing||Corresponding Test Automation|
|Software Engineers (SWE): |
Responsible for coding and test case development. Utilize the “code a little, test a little” method using an Agile Test-Driven Development approach.
|Automatically Generated Tests: |
Provides built-in support for “Test-Driven Development” and allows you to build test cases as soon as the header files are created.
|Software Engineer in Test (SET): |
Assist the SWE with the unit testing. The SET’s are responsible for building test harnesses and frameworks including stubs, mocks, and fakes for every version of the application. The SWE’s use the harnesses to create the tests necessary to fully test the new and modified code.
|Test Harness Generation: |
Replaces the manual effort involved with creating the framework, mocks, stubs, and fakes. Additionally, the tool should automatically update the test artifacts as the code changes.
|Unit Test Dashboard |
Google has created a dashboard to enable them to run every test for every project every day.
|Test Manager/Report Generating Tool: |
In addition to the basic functionality of test control and reporting, it also allows the same tests to be run with different configurations of the source code, or with different compilers.
|Code Coverage Analysis: |
Google analyzes Code Coverage for all levels of testing.
|Code Coverage Metrics Report: |
Capture Statement, Branch, and MC/DC code coverage for any unit, integration, or system level test. It should also allow combining of coverage across all testing into a single metric.
|Dependency Analysis: |
Google has built a dependency analysis of each project so each developer knows exactly what code and test cases are affected by any source change. This provides immediate feedback on how each source change affects the entire system.
|Change-Based Testing Filter: Support for Change-Based Testing, which performs dependency analysis for each source change and computes the sub-set of tests that are directly or indirectly affected by that change. A single click should run only those affected tests, reducing incremental test time from days to minutes.|
Create a continuous integration environment
Developing an all-encompassing software testing environment may seem expensive and daunting at first. Ad hoc testing methods are not beneficial to applications in the long term. At least if they are not augmented to additional testing that is performed.
You should not ignore the benefits of creating a continuous integration environment. You can implement and scale it easily when using off-the-shelf test automation solutions.
Increase software quality
The increase in overall software quality as a result of improved testing enables developers to tackle more features and issues (saving time and cost). It also results in a better end product. The better the product, the better the consumer perception, and the better your business will perform long term.