V&V for Autonomous Vehicles

Today, ECR's Phil Koopman is presenting a paper at SAE World Congress titled "Toward a Framework for Highly Automated Vehicle Safety Validation". One of the most significant challenges to the mass deployment of an autonomous system, the paper and presentation help practitioners some guidance on what they can do to anticipate and mitigate potential hazards and mishaps.

One important part that is implicit in this? V&V is likely to be the single most expensive part of HAV deployment, so optimize your process for V&V.

Abstract:
Validating the safety of Highly Automated Vehicles (HAVs) is a significant autonomy challenge. HAV safety validation strategies based solely on brute force on-road testing campaigns are unlikely to be viable. While simulations and exercising edge case scenarios can help reduce validation cost, those techniques alone are unlikely to provide a sufficient level of assurance for full-scale deployment without adopting a more nuanced view of validation data collection and safety analysis. Validation approaches can be improved by using higher fidelity testing to explicitly validate the assumptions and simplifications of lower fidelity testing rather than just obtaining sampled replication of lower fidelity results. Disentangling multiple testing goals can help by separating validation processes for requirements, environmental model sufficiency, autonomy correctness, autonomy robustness, and test scenario sufficiency. For autonomy approaches with implicit designs and requirements, such as machine learning training data sets, establishing observability points in the architecture can help ensure that vehicles pass the right tests for the right reason. These principles could improve both efficiency and effectiveness for demonstrating HAV safety as part of a phased validation plan that includes both a "driver test" and lifecycle monitoring as well as explicitly managing validation uncertainty.

Article preprint can be found here: http://users.ece.cmu.edu/~koopman/pubs/koopman18_av_safety_validation.pdf