Switchboard: Accelerated Life Testing
Robustness testing finds the unusual "Black Swans" (or, yes, Edge Cases) that are typically only found at scale. No matter how many tests you run on a prototype or small-scale deployment, the real world has a way of presenting scenarios that you can't imagine or can't test for. Switchboard is a cloud-based tool that scales robustness testing, effectively accelerating life testing. It finds those Black Swans before you deploy.
What is robustness testing?
Robustness testing flips traditional software testing on its head. Most software testing tools look for problems in code, whether dealing with code style or other markers. Robustness testing seeks undesirable outcomes ("invariants") and then traces violations of those invariants back to software defects.
This change in perspective means that (1) there are no false positives, and (2) it doesn't try to micromanage your style.
What does switchboard do?
Switchboard is robustness testing taken to the cloud. It takes your system and uses the cloud to scale the speed and coverage of tests.
One implication of this is that using Switchboard is akin to accelerated life testing. System tests are necessary but limited - it is highly unlikely you would find all of the failure modes that can occur in the real world. What Switchboard does is find all the ways the system can fail. This means that you can find the likely failure modes far more quickly and far more accurately than you otherwise could.
Switchboard is a highly configurable tool, compatible with a variety of different systems. Each new system under test (SUT) just needs an interface adapter, and systems that use a standard interface (e.g. ROS) can leverage existing adapters. Even more, once the tool is configured, you can test multiple iterations of your code as you develop it; as long as the interfaces don't change, no further configuration is required.
Who should use switchboard?
At the moment, Switchboard is most compatible with ROS and associated simulators and tools. Gazebo, Simulink, Trucksim, and move_base are examples of tools where we have already done the work of building adapters.
The ECR team has been stress testing complex software for over 20 years.We have found a significant bug - often one that violates a safety-critical requirement - in every system we've ever tested. Examples of System-level vulnerabilities found:
Inappropriate state transitions
Collisions (in simulation!)
Behavior that traces to violations of safety requirements
Loss of control authority
Out-of-bounds control signals
Unacceptably poor responsiveness
Whether dealing with an autonomous vehicle or consumer electronics product, in civilian or defense applications, newly-built code or even mature software, our robustness testing methodologies have been able to find issues that other methodologies don't.