System testing in virtual environments

System testing in virtual environments

When it comes to system level testing, virtual environments offer several strong benefits for you. In this blog post, I want to highlight some of the most important ones:

Early system level tests

Perhaps the biggest benefit is that you can test much earlier on the system level. In a virtual environment, you simply don’t need the target device, and you can start testing right away. This reduces time to market when the actual hardware is simply not available yet. You can also run system level tests alongside with your unit tests. This gives you an early understanding of how your software behaves once it is integrated. And since everything is virtual, it is easy to make these system level part of your CI/CT pipeline, so you can merge changes with high confidence.

Scalable system level tests

When performing system level tests with real hardware, a major limiting factor is the scalability. This is due to the availability of target devices and test equipment. Often, the hardware is still under development and it is even harder to get your hands on a prototype. Thus, test engineers will compete for these resources, which limits the number of tests you can run in parallel. When using virtualization, on the other hand, all of these limitations go away. Every test engineer can create as many virtual environments as needed. You probably still need to run some tests on the target hardware. But you can be sure that the number of these tests is reduced to a minimum.

Simple test specification and execution

Traditional system level tests are hard to reproduce and are prone to errors. Often tests are only specified as a list of manual steps, e.g., in a text document. A test engineer follows these steps, sets up the test equipment and notes down the behavior. Not only is it easy to make a mistake here, but it is also almost impossible to reproduce tests exactly. This is due to the interaction with analog values, e.g., voltages, which always need to account for small variances.

In a virtual environment, you can specify your system level tests as a program. These test programs will apply the exact values every time. And you don’t need to account for variances of analog values because no hardware is involved. Test programs also fit well into DevOps workflows. The tests can be easily executed by your CI/CT pipeline, and you can clearly see how a test changed in a pull request.

Simple fault diagnostics

Hardware debuggers have come a long way and allow you to debug on the target device comfortably. But fully understanding a problem often requires many iterations of debugging. And for each iteration, you have to setup your environment again. When you do this with the target device, resetting the physical environment is a time-consuming task. And, if you are unlucky, it can be hard to hit the breakpoint at the right time. All of this goes away if you rely on virtualization. Not only is it easy to reset your environment. But you can also pause the execution of the environment, making debugging a much easier task.

Independence of back-end services availability

These days, many embedded devices are smart devices that require back-end services for proper operation. To setup a back-end service for testing can be a complex task. E.g., you have to clean and fill the underlying database for every test. This becomes easier if you virtualize the service with a model. The benefit of models is that they guarantee reproducible results. They also do not need a live internet connection, which might not be available otherwise.

System testing in critical or fatal scenarios

Last but not least a benefit that is not obvious but even more so important. Ensuring that your devices operate correctly in critical or fatal scenarios is often hard or costly. For example, consider that your device has to safely shut down if a motor breaks. To emulate such a motor break in the real world is at least a costly task. But in a virtual environment, it can simply be a feature of your model. There are many more situations that are hard to emulate in the real world but easy in the virtual one. E.g., testing input values out of the specified ranges and ensuring that your device does not do harm.

How to virtualize your system tests

Details about system testing in virtual environments

Learn more in our video. In this video, we show how to virtualize all aspects of your system level tests. This includes:

  1. Your system under test
  2. The physical environment
  3. The software environment

Further readings / information:


Legal Notice

This text is the intellectual property of the author and is copyrighted by You are welcome to reuse the thoughts from this blog post. However, the author must always be mentioned with a link to this post!

Leave a Comment

Related Posts

Daniel Lehner

Software Testing in the World of IoT

This interview answers the question “How can I reduce the testing effort for the software of my IoT system?” The expert from Vector explains, how

Daniel Lehner

Reducing efforts in system testing

Which are the best methods to reduce efforts in system testing? In this second part of the Coders Kitchen interview, system testing experts Hans Quecke

Traditional system testing vs. model-based system testing - Coders Kitchen
Daniel Lehner

Traditional vs. model-based system testing

Interview with system testing experts Simone Gronau and Hans Quecke about typical challenges in defining system tests, and how model-based methods might offer an alternative.

Hey there!

Subscribe and get a monthly digest of our newest quality pieces to your inbox.