Robustness testing

Robustness testing finds the unusual "Black Swans" that are typically only found at scale - after you have deployed 1000 or 10,000 units. No matter how many tests you run on a prototype or small-scale deployment, the real world has a way of presenting scenarios that you can't imagine or can't test for.

Robustness testing flips traditional software testing on its head. Most software testing tools look for problems in code, whether dealing with code style or other markers. In addition, traditional software testing methods are often functional tests – “does the software work” vs. “what breaks the system”?

runtime invariants.png

Robustness testing seeks specified undesirable outcomes ("invariants") and then traces violations of those invariants back to software defects. This technique makes it especially useful in finding edge cases.

WHY IS THAT IMPORTANT?

An  example of ECR's workflow

No matter how many tests you run on a prototype or small-scale deployment, the real world has a way of presenting scenarios that you can't imagine or can't test for. So rather than trying to test for the potential infinite number failure modes you and your team can try to imagine, test for undesirable outcomes.

Think of an autonomous robotics system: it’s one thing to determine that a system can work in a single, well-defined environment. However, when you take a system to an unbounded (or differently-bounded) environment, you may need to go back to the drawing board to address those changes. Finding edge cases helps to make systems more adaptable, and scalable, especially in safety-critical contexts.

EDGE CASE RESEARCH
100 43RD ST.
SUITE 208
PITTSBURGH, PA 15201

Copyright 2015, Edge Case Research, LLC. All rights reserved.