Phil Koopman's comments on automated vehicle safety in the Post-Gazette
In an Op-Ed in the Pittsburgh Post-Gazette, ECR Chief Scientist Dr. Philip Koopman discussed the challenges in evaluating autonomous vehicle safety. In part, the op-ed is a response to the recent draft policy set forth by the US DoT and NHTSA, released earlier in September 2016. To be clear, Koopman believes that the draft policy is an enormous leap forward in codifying this important area, but still has some issues that need to be addressed.
In the op-ed, Koopman talks about two key issues. First, the challenges in verification and validation of machine learning technologies.
Traditional software safety checks the software’s recipe against its actual behavior. But with machine learning there is no recipe – just a huge bunch of examples. So it is difficult to be sure how the car will behave when it sees something that differs even slightly from an example. As an overly-simplistic example, perhaps a car will stop for pedestrians in testing because it has learned that heads are round, but has trouble with unusual hat shapes.
Second, the principle of independence, making those professionals responsible for safety separate from those who are responsible for the design and release of the vehicle. This is common practice every safety-critical field...aside from cars.
In airplanes, trains, medical devices and even UL-listed appliances, ensuring safety requires an independent examination of software design. This helps keep things safe even when management feels pressure to ship a product before it’s really ready.
In addition, balancing the need for routine software software updates vs the need to go through an approval process for safety-critical software also needs to be addressed in the DoT's policy. To wit:
The policy should be changed to require a safety certification for every software change that potentially affects safety. This sounds like a lot of work, but designers have handled this for decades by partitioning vehicle software across multiple independent computers. Changing an icon on the radio computer’s display? No problem. Making a “minor” change to the machine learning data for the pedestrian sensor computer? Sorry, but I want an independent safety check on that before I cross the street in my winter hat.