SAE 2016: Challenges in Autonomous Vehicle Testing and Validation

A generic V model.

ECR Co-Founders Philip Koopman and Michael Wagner presented their vision on the future of software testing to create safe autonomous vehicles at the SAE World Congress in Detroit.

In this paper Koopman and Wagner explore some of the challenges that await developers who are attempting to qualify fully autonomous, NHTSA Level 4 vehicles for large-scale deployment.


Software testing is all too often simply a bug hunt rather than a well-considered exercise in ensuring quality. A more methodical approach than a simple cycle of system-level test-fail-patch-test will be required to deploy safe autonomous vehicles at scale. The ISO 26262 development V process sets up a framework that ties each type of testing to a corresponding design or requirement document, but presents challenges when adapted to deal with the sorts of novel testing problems that face autonomous vehicles. This paper identifies five major challenge areas in testing according to the V model for autonomous vehicles: driver out of the loop, complex requirements, non-deterministic algorithms, inductive learning algorithms, and fail-operational systems. General solution approaches that seem promising across these different challenge areas include: phased deployment using successively relaxed operational scenarios, use of a monitor/actuator pair architecture to separate the most complex autonomy functions from simpler safety functions, and fault injection as a way to perform more efficient edge case testing. While significant challenges remain in safety-certifying the type of algorithms that provide high-level autonomy themselves, it seems within reach to instead architect the system and its accompanying design process to be able to employ existing software safety approaches.

You can get the full paper here.

100 43RD ST.

Copyright 2015, Edge Case Research, LLC. All rights reserved.