Accelerated Life Testing
Robustness testing finds the unusual "Black Swans" (or, yes, Edge Cases) that are typically only found at scale. No matter how many tests you run on a prototype or small-scale deployment, the real world has a way of presenting scenarios that you can't imagine or can't test for. Switchboard is a cloud-based tool that scales robustness testing, effectively accelerating life testing. It finds those Black Swans before you deploy.
What is Robustness Testing?
Robustness testing flips traditional software testing on its head. Most software testing tools look for problems in code, whether dealing with code style or other markers. In addition, traditional software testing methods are often functional tests – “does the software work” vs. “what breaks the system”?
Robustness testing seeks specified undesirable outcomes and then traces violations of those invariants back to software defects. This technique makes it especially useful in finding edge cases.
Why is that important?
No matter how many tests you run on a prototype or small-scale deployment, the real world has a way of presenting scenarios that you can't imagine or can't test for.
Think of an autonomous robotics system: it’s one thing to determine that a system can work in a single, well-defined environment. However, when you take a system to a new – even slightly different – environment, you may need to go back to the drawing board to address those changes. Finding edge cases helps to make systems more adaptable, and scalable, even in safety-critical contexts.
Switchboard for ROS is robustness testing taken to the cloud. It takes your system and uses the cloud to scale the speed and coverage of tests.
One implication of this is that using Switchboard is akin to accelerated life testing. System tests are necessary but limited - it is highly unlikely you would find all of the failure modes that can occur in the real world. What Switchboard does is find all the ways the system can fail. This means that you can find the likely failure modes far more quickly and far more accurately than you otherwise could.
It also helps to make bespoke robotic deployments more flexible, and by extension, scalable.
How does it work?
Switchboard for ROS is a highly configurable, parallelizable tool, compatible with a variety of different systems. Each new system under test (SUT) just needs an interface adapter, and systems that use a standard interface (e.g. ROS) can leverage existing adapters. Even more, once the tool is configured, you can test multiple iterations of your code as you develop it; as long as the interfaces don't change, no further configuration is required.
Once set up, it is possible to run as many tests as you want, over as long or short a time frame as you wish or your budget allows.
Anyone who uses ROS and its associated tools can use Switchboard for ROS. Gazebo, Simulink, Trucksim, and move_base are examples of tools where we have already done the work of building adapters for.
Who Should use it?
If you intend to deploy autonomous systems at scale, you should consider using Switchboard for ROS. In addition, even if you are considering a smaller-scale deployment, Switchboard for ROS is an invaluable too for anyone who is deploying robotics in a safety- or mission-critical context.
Why you should trust ECR
The ECR team has been stress testing complex software for over 20 years. We have found a significant bug - often one that violates a safety-critical requirement - in every system we've ever tested. We don’t test wimpy software.
Whether dealing with an autonomous vehicle or consumer electronics product, in civilian or defense applications, newly-built code or even mature software, our robustness testing methodologies have been able to find issues that other methodologies don't.