When it comes to system level testing, virtual environments offer several strong benefits for you. In this blog post, I want to highlight some of the most important ones:

Early system level tests

Perhaps the biggest benefit is that you can test much earlier on the system level. In a virtual environment, you simply don’t need the target device, and you can start testing right away. This reduces time to market when the actual hardware is simply not available yet. You can also run system level tests alongside with your unit tests. This gives you an early understanding of how your software behaves once it is integrated. And since everything is virtual, it is easy to make these system level part of your CI/CT pipeline, so you can merge changes with high confidence.

Scalable system level tests

When performing system level tests with real hardware, a major limiting factor is the scalability. This is due to the availability of target devices and test equipment. Often, the hardware is still under development and it is even harder to get your hands on a prototype. Thus, test engineers will compete for these resources, which limits the number of tests you can run in parallel. When using virtualization, on the other hand, all of these limitations go away. Every test engineer can create as many virtual environments as needed. You probably still need to run some tests on the target hardware. But you can be sure that the number of these tests is reduced to a minimum.

Simple test specification and execution

Traditional system level tests are hard to reproduce and are prone to errors. Often tests are only specified as a list of manual steps, e.g., in a text document. A test engineer follows these steps, sets up the test equipment and notes down the behavior. Not only is it easy to make a mistake here, but it is also almost impossible to reproduce tests exactly. This is due to the interaction with analog values, e.g., voltages, which always need to account for small variances.

In a virtual environment, you can specify your system level tests as a program. These test programs will apply the exact values every time. And you don’t need to account for variances of analog values because no hardware is involved. Test programs also fit well into DevOps workflows. The tests can be easily executed by your CI/CT pipeline, and you can clearly see how a test changed in a pull request.

Simple fault diagnostics

Hardware debuggers have come a long way and allow you to debug on the target device comfortably. But fully understanding a problem often requires many iterations of debugging. And for each iteration, you have to setup your environment again. When you do this with the target device, resetting the physical environment is a time-consuming task. And, if you are unlucky, it can be hard to hit the breakpoint at the right time. All of this goes away if you rely on virtualization. Not only is it easy to reset your environment. But you can also pause the execution of the environment, making debugging a much easier task.

Independence of back-end services availability

These days, many embedded devices are smart devices that require back-end services for proper operation. To setup a back-end service for testing can be a complex task. E.g., you have to clean and fill the underlying database for every test. This becomes easier if you virtualize the service with a model. The benefit of models is that they guarantee reproducible results. They also do not need a live internet connection, which might not be available otherwise.

System testing in critical or fatal scenarios

Last but not least a benefit that is not obvious but even more so important. Ensuring that your devices operates correctly in critical or fatal scenarios is often hard or costly. For example, consider that your device has to safely shut down if a motor breaks. To emulate such a motor break in the real world is at least a costly task. But in a virtual environment, it can simply be a feature of your model. There are many more situations that are hard to emulate in the real world but easy in the virtual one. E.g., testing input values out of the specified ranges and ensuring that your device does not do harm.

How to virtualize your system tests

Details about system testing in virtual environments

Learn more in our video. In this video, we show how to virtualize all aspects of your system level tests. This includes:

  1. Your system under test
  2. The physical environment
  3. The software environment

Further readings / information:


Share on linkedin
Share on twitter
Share on whatsapp

Leave a Comment

Related Posts

Martin Heininger

Testing AI-based systems

AI-based systems are becoming popular. However, the broad acceptance of such systems is still quite low in our society. Trust in and acceptance of AI-based

Improving test efficiency
John Paliotta

Improving test efficiency

Effective software teams are always looking for ways to make their members more efficient. They realize that improving the development process is an good way

Debugging without Security risks
Niroshan Rajadurai

Debugging without security risks

Security researcher Michael Myng found the keylogging code in software drivers that were preinstalled on HP laptops to make the keyboard work. He discovered the

Quantifying the cost of fixing bugs
Lynda Gaines

Cost of fixing vs. preventing bugs

When you think about improving software quality, your initial thoughts might be the cost of investing in new tools and engineering labor to implement them.

Error: Safety security both
Sebastian Krings

A simple error: Safety, security or both?

Aside from safety properties, can we use static analysis tools to detect security issues? Yes, as we will show by discussing a programming error in uftpd,

Hey there!

Subscribe and get a monthly digest of our newest quality pieces to your inbox.