V&V for Autonomous Vehicles

Today, ECR's Phil Koopman is presenting a paper at SAE World Congress titled "Toward a Framework for Highly Automated Vehicle Safety Validation". One of the most significant challenges to the mass deployment of an autonomous system, the paper and presentation help practitioners some guidance on what they can do to anticipate and mitigate potential hazards and mishaps.

One important part that is implicit in this? V&V is likely to be the single most expensive part of HAV deployment, so optimize your process for V&V.

Validating the safety of Highly Automated Vehicles (HAVs) is a significant autonomy challenge. HAV safety validation strategies based solely on brute force on-road testing campaigns are unlikely to be viable. While simulations and exercising edge case scenarios can help reduce validation cost, those techniques alone are unlikely to provide a sufficient level of assurance for full-scale deployment without adopting a more nuanced view of validation data collection and safety analysis. Validation approaches can be improved by using higher fidelity testing to explicitly validate the assumptions and simplifications of lower fidelity testing rather than just obtaining sampled replication of lower fidelity results. Disentangling multiple testing goals can help by separating validation processes for requirements, environmental model sufficiency, autonomy correctness, autonomy robustness, and test scenario sufficiency. For autonomy approaches with implicit designs and requirements, such as machine learning training data sets, establishing observability points in the architecture can help ensure that vehicles pass the right tests for the right reason. These principles could improve both efficiency and effectiveness for demonstrating HAV safety as part of a phased validation plan that includes both a "driver test" and lifecycle monitoring as well as explicitly managing validation uncertainty.

Article preprint can be found here: http://users.ece.cmu.edu/~koopman/pubs/koopman18_av_safety_validation.pdf


ECR at the 2018 PAAV Summit


Phil Koopman will be giving the keynote address on Monday, April 9 at 9:15am and will discuss "Three Key Questions for Self-Driving Cars on Public Roads". He will get into topics related to the safe testing of autonomous vehicles and what autonomous vehicle makers, policymakers, and regulators can do to ensure public safety while still allowing the development of this important technology.

Immediately following the keynote, Mike Wagner will participate in the "Ensuring Automated Vehicle Safety" panel at 9:45am. The panel will focus on if and how autonomous vehicle testing and operation can and should be made safe. In addition, it will explore what role makers, regulators, and other stakeholders like law enforcement will need to have in the development of safe autonomous vehicles.

Event Details:

PAAV Summit 2018
Agenda: https://www.paavsummit.org/agenda.html

April 9-10, 2018

DoubleTree by Hilton Hotel & Suites
One Bigelow Square
Pittsburgh, PA 15219


Edge Case Research Collaborates with Kaarta to Accelerate Autonomous Vehicle Safety

ECR delivers on commitment to safety with Hologram, an intelligent testing platform to prepare autonomous cars for the road.

Edge Case Research (ECR), a leader in autonomous vehicle safety, is collaborating with Kaarta, innovator of real-time 3D reality capture, to revolutionize autonomous vehicle road testing. ECR has kicked off an initiative to map Pittsburgh’s streets using Kaarta 3D mapping and localization technology. ECR will use these maps to pilot its Hologram platform, a tool that maximizes the value of every mile driven by transforming each recorded scenario into a million realistic potential scenarios to test. Hologram intelligently and quickly identifies issues, enabling manufacturers to significantly improve testing and time to market for safe autonomous vehicles.

Autonomous vehicle companies are currently reporting miles driven and number of disengagements as metrics to demonstrate safety; however, not all miles are equal. Even billions of simulation miles won't necessarily expose cars to life-threatening situations. The industry demands a tool to find edge cases more quickly and cost-effectively than methods in use today.

Hologram consumes data recorded during field testing. From this real-world data, Hologram generates countless potential edge-case scenarios by adjusting a multitude of variables such as lighting, weather effects, the location of cars and pedestrians amongst others. By combining these factors, Hologram automatically produces and tests limitless realistic scenarios in order to validate the safety of autonomous perception systems.

“Safe perception is critical for safe autonomous vehicles, and that’s why we’re building Hologram,” said Edge Case Research, CEO, Michael Wagner. “Hologram’s ability to create millions of unique test cases in a given street or environment means that developers of perception systems can train and validate their machine learning systems far more efficiently.” 

Using Kaarta technology to collect data of Pittsburgh’s bustling downtown, Hologram has successfully identified cases in which leading pedestrian-detection algorithms lose track of people crossing the street. The City of Pittsburgh’s unique and challenging urban environment is the only urban location designated by the U.S. Department of Transportation as an automated vehicle proving ground, making it an ideal location for piloting Hologram. 

“Kaarta’s mapping technology turns raw sensor data into intelligent 3D environment models in real time,” said Kaarta CEO, Kevin Dowling. “These models provide high dimensional fidelity of the real world for immediate testing. This work ECR and Kaarta are doing is proving to be very fruitful for both companies and for the autonomous vehicle industry as a whole.”

The technology partnership of ECR and Kaarta provides unparalleled value to the safety of the AV industry. Additionally, ECR can make the 3D maps generated during Hologram development available to city planners and public stakeholders in need of highly accurate models.   


About Edge Case Research:

The future depends on AI, and Edge Case Research makes it safe by delivering software and services to tackle the most complex AI safety problems. Our customers demand tools that help them go to market with products that they know are safe. Edge Case Research was formed by leading autonomy and safety experts from Carnegie Mellon University who believe that safety should be built into all AI products from the ground up. Our intelligent and fast robustness testing platforms find safety problems in autonomous vehicles before they are fielded. Our growing client base spans diverse markets including automotive, aerospace, defense, consumer electronics, and industrial power systems. Join the safety movement. See more information at our website: www.edge-case-research.com.


About Kaarta:

Kaarta is the innovator of real-time 3D reality capture, mapping complex environments with unprecedented speed, fidelity and ease. Kaarta’s patent-pending technology, rooted in advanced robotics, accurately transforms the real world into multi-dimensional 3D digital models, streamlining workflow, reducing time, lowering cost, and enabling entirely new applications. Now the spectrum of stakeholders across architecture, engineering, construction, property management, surveying, and facilities management can work from the same truth, going from silos to integrated teams in the design, build, and operations of the built environment. Additional information is available at www.kaarta.com.

Media Contact:
Jackie Erickson