Automate 2017: Collaborative Robotic Safety

One of the key challenges to the use of machine/deep learning systems has been in testing and validating them to ensure safety. Our co-founders have written a paper on this topic as it relates to autonomous vehicles in the past, but the fundamental challenge is relevant to any autonomous system, including collaborative robotics. That's why our CEO, Mike Wagner, will be presenting at the "Collaborative Robots: Safety Implications" panel at Automate 2017. The panel will also include Carole Franklin of the Robotics Industry Association and Thomas Knauer of Omron STI. The panel will be held from 3pm to 5pm on Tuesday, April 4.

Brittleness in deep learning perception and pattern recognition is one of the challenges to creating a safe system.

Brittleness in deep learning perception and pattern recognition is one of the challenges to creating a safe system.

Mike will discuss some of the challenges in validating safety in collaborative robotics, particularly when it involves systems that incorporate deep learning. Mike will talk about how we approach safety when traditional safety practices are frustrated, and the techniques and tools we use to help organizations overcome those challenges.

 

 

 

Safety is not just an engineering challenge

IEEE Transportation Systems Magazine's Spring 2017 issue had a theme of safety and security for autonomous vehicles and co-founders, Mike Wagner and Phil Koopman were asked to weigh in. 

The short, short version of the article:

  • Understanding what "safe" even means in the context of an autonomous system is harder than it sounds, and requires cross-domain expertise.
  • Validating inductive reasoning (as machine learning systems do) is inherently difficult and has been known to be at least since David Hume's time.
  • Safe autonomy can be accomplished and deployed, but safety certification that incorporates input from safety engineers, security experts, software validation experts, HCI experts, a viable legal framework, and many others will have to meaningfully contribute to its structure. 

Abstract:

Ensuring the safety of fully autonomous vehicles requires a multi-disciplinary approach across all the levels of functional hierarchy, from hardware fault tolerance, to resilient machine learning, to cooperating with humans driving conventional vehicles, to validating systems for operation in highly unstructured environments, to appropriate regulatory approaches. Significant open technical challenges include validating inductive learning in the face of novel environmental inputs and achieving the very high levels of dependability required for full-scale fleet deployment. However, the biggest challenge may be in creating an end-to-end design and deployment process that integrates the safety concerns of a myriad of technical specialties into a unified approach.

Article pre-print can be found here

Five Forebodes Failure

First shalt thou take out the Holy Pin, then shalt thou count to three, no more, no less. Three shall be the number thou shalt count, and the number of the counting shall be three. Four shalt thou not count, neither count thou two, excepting that thou then proceed to three. Five is right out. Once the number three, being the third number, be reached, then lobbest thou thy Holy Hand Grenade of Antioch towards thy foe, who being naughty in My sight, shall snuff it.
— from Monty Python and the Holy Grail

There are lots of reasons that embedded software projects fail. Informal development processes, lack of requirements (or poor requirements), coding errors, undefined software architecture, developer or engineer turnover, and so on. There is, however, one anecdotal but surprisingly consistent finding that makes us scratch our heads; developer teams adding the 5th primary contributor often have spectacular project failures.

One thing that we have noticed from both design reviews across teams in a variety of industries as well as in classroom activities: when you have a team of four on a software development project and add one more member… the project tends to fall apart. Three developers on a specific project works fine. Usually the fourth person added to a team helps, although potentially with decreased productivity contribution. (For student teams our data shows that the fourth person doesn’t add any extra productivity, but they’re still learning and only have a few hours per week to spend on projects for a particular class.) However, once a team reaches five individuals, the fifth person, no matter their qualifications or expertise, usually contributes negative productivity, and often causes a project failure.

You would think that adding a fifth member to a team results in at least a marginal productivity increase, and perhaps that happens in some cases. But time after time, we’ve seen that adding a fifth member to the team not only doesn’t add any productivity, but it is often counterproductive.

Why does one more team member often make such a big (bad) difference?

There is no one-size-fits-all answer, but we do have one hypothesis: most teams that add person #5 have weak or excessively informal processes, and are adding that person expecting to get linear productivity improvement. When you have 3 or 4 developers, you can often get away with a weak processes, especially if you have a stable team and work on a single product family over a period of years. 3 or 4 developers in a shared office area can form a tight group and coordination overhead is low enough to keep everyone on the same page. However, if you have a project that gets so complex that 5 developers need to coordinate on that same project, you have reached a tipping point. Informal processes and communication break down, and you often have a spectacular process failure.

Typically, teams that have a more formal, rigorous development process are less likely to suffer from the “Five Forebodes Failure” phenomenon, because they’re already spending the equivalent of most or all of that 5th person of effort on explicit coordination and software development process activities. Often the team leader is spending perhaps 10 hours per week on pure process support. (By the time you get to a team of 20 you’ll need a full time software process and SQA person.) Teams of six or more working on a project often have a more formalized process, or have been very clever at breaking a project down into numerous almost-completely independent components that don’t require much coordination.

All of this isn’t to say that adding a fifth team member to a project immediately means that you’re going to fail. But it does indicate that a good heuristic would be to tread carefully when you get to a team size of 4-6 developers, and expect to have to add more formal processes, especially if your current development process is informal.

We’re willing to bet you’re not keen on spending a whole 5th person on process. (If you want to add person #5, you probably are doing that so they can write code.) The key to success with developer #5 is not to go crazy with overly-burdensome paperwork, but rather to make sure you have some key process steps formalized to keep things under control.

We have a one-day on-site seminar that can teach you the top ways to improve software quality and process quality in a way that provides just enough process support to keep things from getting crazy while remaining productive. Contact us for more information.

References:

Koopman, 2010: "Risk Areas In Embedded Software Industry Projects" 

ECR Media Mentions: NPR, embedded.fm

There’s actually often a bias that we see where a developer will think that a certain scenario just won’t happen— ‘Oh, that bug you found will never occur in practice.’ [But] if you have a little bit of gray hair, you realize that these things do occur.
— Mike Wagner, CEO of Edge Case Research

First, we had a chance to talk a bit about how our company helps improve the software that powers autonomous robotics, vehicles, and other embedded systems. Mike Wagner was interviewed by WESA and NPR's Mark Nootbaar (@WESAmark). Mike talked a bit about how Edge Case Research's robustness testing methodology helps companies deploy robust software: reliable, safer, and secure code. You can find the streaming audio and accompanying article here.  

Second, Phil Koopman (@BetterEmbSW - yes, he's joined the dark side) had the chance to appear on the Embedded.fm podcast, hosted by Elicia (@logicalelegance) and Chris (@stoneymonster) White. If you have strong opinions about Agile development in embedded, or paper airplanes, or breaking robots, you may want to take a listen. Also, if you are at all interested in embedded systems, best practices, and peer reviews - yup, that definitely comes up - then this will be well worth your time.

Self-Driving Cars Test Traditional Procedures - Digital Engineering

This article by Beth Stackpole is a very good overview of the challenges of validating autonomous vehicle safety - and not just because we were interviewed for it and quoted in it. The article goes into the various challenges to autonomous vehicle testing and validation, something we've wrote about before.

One important addition that is outside of the scope of the article: it's not just a challenge for autonomous vehicles.  Pretty much every safety- and mission-critical embedded software system is being challenged with complexity and uncertainty. Simulation tools are an important piece of ensuring that the deployment of such vehicles are fast, safe, and secure.

We are building a scalable solution to address this complexity based on our 20 years of robustness testing experience.  This was briefly mentioned in the article, that we will discuss more openly soon.

Article link: http://www.digitaleng.news/de/self-driving-cars-test-traditional-procedures/

You have the potential for millions of combinations that the algorithm may not have learned yet, and each represents a gap in the knowledge of the pedestrian detector, a software error, and a potential safety hazard.
— Michael Wagner, CEO and founder of Edge Case Research
What we’re doing at a software level is addressing the stuff that doesn’t happen often to make sure the system doesn’t trip when it sees something weird. This kind of testing provides confidence that all the failure modes that might happen have been explored … and it’s helpful to pair [the practice] with simulation testing.
— Phil Koopman, ECR co-founder and Chief Technical Officer

Mike Wagner to speak on autonomous vehicle safety at AUVSI’s Unmanned Systems Defense 2016 Conference

On Thursday, October 27, CEO Mike Wagner will be speaking at the AUVSI Unmanned Systems Defense conference in Arlington, VA. Unmanned Systems Defense is a conference that intends to bring together government program managers, decision makers, and technology experts for three intensive days of information sharing and interaction. At the conference, Wagner will speak as part of a panel on the Future of Ground Robot Autonomy; the panel will focus on safety challenges for autonomous robotic safety in particular.

Software testing is all too often simply a "bug hunt" rather than a well-considered exercise in ensuring quality. We need more than a simple cycle of system-level "test-fail-patch-test" to deploy safe autonomous vehicles into critical defense applications. Traditional software-safety processes tie each type of testing to a corresponding design or requirement document; however, these processes face challenges when applied to algorithms necessary for autonomous systems.

We have identified several major challenge areas in testing, and will discuss promising potential solutions. While significant challenges remain in safety-certifying the types of algorithms used to achieve artificial intelligence, it seems within reach to instead architect such a system to support existing software safety approaches.

If you are interested to see how the (ever-closer) future of autonomous robotics will evolve and what can be done to address the complex challenges involved in developing safe, secure systems, you will want to attend.

Embedded software quality best practices video library is now live!

I went to my first computer conference at the New York Hilton about 20 years ago. When somebody there predicted the market for microprocessors would eventually be in the millions, someone else said, “Where are they all going to go? It’s not like you need a computer in every doorknob!” Years later, I went back to the same hotel. I noticed the room keys had been replaced by electronic cards you slide into slots in the doors. There was a computer in every doorknob.
— Danny Hillis

One of the core missions Edge Case Research is to help individuals and organizations to develop safe, secure, and high-quality embedded code. Developing high-quality embedded software is extremely difficult, no matter the application, industry, or the perceived simplicity of the code.

Computers and microcontrollers are now everywhere, and embedded software doubly so. Despite this proliferation, not everyone who is actively developing software has had formal training in software development. Co-founder and Chief Technologist Phil Koopman has put together a number of videos on common problems in embedded software development and what you can do to address them.

To begin with, there are three sets of videos: Peer Reviews, Spaghetti Code, and Security Planning. Each has a short preview and a full tutorial on how to implement the practice or address the issue.

To see the videos, you can click here or go to http://www.edge-case-research.com/videos.

Note that you can also find more videos on our Vimeo channel as well as our Youtube channel.

Edge Case Research (and many, many others) celebrate robotics in Pittsburgh

Our CEO at the table along with a cup of what makes everything possible. Thanks, coffee!

Our CEO at the table along with a cup of what makes everything possible. Thanks, coffee!

It's rarely been more apparent that the Steel City has become a technology center than today at the Carnegie Robotics high bay. 21 local robotics companies came together to celebrate and promote one of the fastest-growing industries in the area.

Also there to share in the festivities were US Representative Mike Doyle and Allegheny County Executive Rich Fitzgerald. CMU Professor and Near Earth Robotics CEO Sanjiv Singh and Carnegie Robotics CEO Steve DiAntonio also gave their remarks.

We'd like to thank our neighbors Carnegie Robotics for hosting the event as well as the Pittsburgh Robotics Network and Jackie Erickson for putting everything together so quickly.

EDGE CASE RESEARCH
100 43RD ST.
SUITE 208
PITTSBURGH, PA 15201

Copyright 2015, Edge Case Research, LLC. All rights reserved.