Incident: Self-Driving Taxi Goes Rogue Due to Remote Guidance Error

Published Date: 2021-05-15

Postmortem Analysis
Timeline 1. The software failure incident with the self-driving taxi happened last Wednesday, as mentioned in the article [114925]. 2. Published on 2021-05-15. 3. The incident occurred in May 2021.
System The system that failed in the software failure incident reported in Article 114925 was: 1. Waymo's remote Fleet Response team provided incorrect guidance, causing the self-driving Chrysler van to have difficulties in resuming its intended route, ultimately requiring Waymo's Roadside Assistance team to complete the trip [114925].
Responsible Organization 1. The mistake made by a remote Fleet Response specialist was responsible for causing the software failure incident [114925].
Impacted Organization 1. Technicians who arrived to help the self-driving taxi [114925] 2. Other drivers on the road who were affected by the self-driving taxi's erratic behavior [114925]
Software Causes 1. The software failure incident in the self-driving taxi incident was caused by incorrect guidance provided by a remote Fleet Response specialist, making it challenging for the Waymo Driver to resume its intended route [114925].
Non-software Causes 1. The self-driving van got confused by traffic cones, leading to the initial blockage and confusion [114925]. 2. The remote Fleet Response specialist provided incorrect guidance, making it challenging for the Waymo Driver to resume its intended route [114925]. 3. The safety driver of a self-driving Uber SUV was distracted, resulting in an accident that led to the first recorded death involving a self-driving car [114925].
Impacts 1. The self-driving taxi caused a partial-traffic blockage, leading to disruption and inconvenience for other drivers on the road [114925]. 2. The incident resulted in confusion and potential danger as the self-driving taxi made unexpected movements, almost colliding with passing vehicles [114925]. 3. The software failure incident led to a delay in the arrival of roadside assistance, prolonging the situation and the blockage caused by the self-driving van [114925]. 4. The incident highlighted a flaw in the remote assistance provided by Waymo, where incorrect guidance from the Fleet Response team contributed to the challenges faced by the self-driving vehicle [114925]. 5. The software failure incident showcased the limitations of remote assistance in taking direct control of the self-driving vehicles, as the team could only offer suggestions rather than direct intervention [114925].
Preventions 1. Implementing better training and protocols for the remote Fleet Response specialists to provide accurate guidance to the self-driving vehicles in challenging situations could have prevented the software failure incident [114925]. 2. Enhancing the communication and coordination between the remote assistance team and the self-driving vehicles to ensure prompt and effective support in case of software glitches or confusion could have helped prevent the incident [114925]. 3. Conducting thorough testing and simulations to identify and address potential software bugs or unexpected behaviors in the self-driving system could have preemptively prevented such incidents [unknown].
Fixes 1. Implement stricter protocols and training for remote Fleet Response specialists to ensure they provide correct guidance to self-driving vehicles in challenging situations [114925]. 2. Enhance the communication and coordination between the remote assistance team and the self-driving vehicles to prevent similar incidents in the future [114925]. 3. Conduct thorough reviews and assessments of software incidents to identify areas for improvement and implement necessary changes to enhance the operational process of self-driving vehicles [114925].
References 1. YouTube video captured by Joel Johnson, YouTuber of 'JJ Ricks Studio' [Article 114925] 2. Statement from Waymo regarding the incident [Article 114925] 3. The Verge report on Waymo's self-driving vehicles and remote assistance team [Article 114925] 4. BBC News report on the first death involving a self-driving car and Uber's response [Article 114925]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident happened again at one_organization: - Waymo, the company behind the self-driving taxi involved in the incident, had a previous incident where a mistake made by a remote Fleet Response specialist led to incorrect guidance, making it challenging for the Waymo Driver to resume its intended route [114925]. (b) The software failure incident happened again at multiple_organization: - The article does not mention any similar incidents happening at other organizations or with their products and services.
Phase (Design/Operation) design, operation (a) The software failure incident in the article was related to the design phase. Waymo stated that the incident captured by Joel Johnson was caused by a mistake made by a remote Fleet Response specialist, who provided incorrect guidance that made it challenging for the Waymo Driver to resume its intended route. This mistake required Waymo's Roadside Assistance team to complete the trip, indicating a failure due to contributing factors introduced by the system development or operational procedures [114925]. (b) The software failure incident in the article was also related to the operation phase. The incident involved the self-driving Chrysler van getting confused by traffic cones, stopping abruptly, blocking the road, and driving off before roadside assistance could arrive. This sequence of events highlights failures introduced by the operation or misuse of the self-driving system [114925].
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident involving the Waymo self-driving taxi was primarily caused by a mistake made by a remote Fleet Response specialist who provided incorrect guidance, making it challenging for the Waymo Driver to resume its intended route [114925]. This indicates that the failure originated from within the system, specifically from the actions of the remote Fleet Response specialist. (b) outside_system: The incident was also influenced by external factors such as the unexpected blockage on the road that confused the self-driving van, leading to it getting stuck and requiring roadside assistance [114925]. Additionally, the presence of construction cones and the interaction with other drivers on the road contributed to the chaotic situation, showing that factors external to the system played a role in the software failure incident.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident in the article was primarily due to non-human actions. The incident occurred when a self-driving taxi operated by Waymo got confused by traffic cones, leading to it stopping and causing a partial traffic blockage. The remote Fleet Response team provided incorrect guidance, making it challenging for the self-driving vehicle to resume its intended route, ultimately requiring Waymo's Roadside Assistance team to intervene and complete the trip [114925]. (b) Human actions also played a role in the incident. The YouTuber, Joel Johnson, captured the entire incident and was in communication with a remote operator during the chaotic ride. Additionally, the construction man asked Johnson to move the car as it was blocking the lane, and Johnson interacted with the operator when the car abruptly took off before technicians could reach it [114925].
Dimension (Hardware/Software) software (a) The software failure incident in the article was not directly attributed to hardware issues. The incident was caused by a mistake made by a remote Fleet Response specialist, which led to incorrect guidance being provided to the self-driving vehicle, making it challenging for the Waymo Driver to resume its intended route [114925]. (b) The software failure incident in the article was primarily attributed to contributing factors originating in software. Waymo stated that the incident captured by Joel Johnson was caused by a mistake made by a remote Fleet Response specialist, which hindered the Waymo Driver from resuming its intended route and required intervention from Waymo's Roadside Assistance team to complete the trip. This indicates that the failure was related to software issues rather than hardware problems [114925].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident described in the article is non-malicious. The incident occurred when a self-driving taxi operated by Waymo got confused by traffic cones, leading to a series of unexpected behaviors such as stopping abruptly, blocking traffic, and driving off before roadside assistance could arrive. The failure was attributed to incorrect guidance provided by a remote Fleet Response specialist, which made it challenging for the self-driving vehicle to resume its intended route [114925].
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident involving the Waymo self-driving taxi was related to poor decisions made by a remote Fleet Response specialist. The incident was caused by incorrect guidance provided by the Fleet Response team, which made it challenging for the Waymo Driver to resume its intended route, ultimately requiring Waymo's Roadside Assistance team to complete the trip [114925]. This poor decision led to the confusion and difficulties faced by the self-driving vehicle during the incident.
Capability (Incompetence/Accidental) accidental (a) The software failure incident in the article was not directly attributed to development incompetence. However, it was mentioned that the incident was caused by a mistake made by a remote Fleet Response specialist who provided incorrect guidance, making it challenging for the self-driving vehicle to resume its intended route [114925]. (b) The software failure incident in the article was more aligned with an accidental failure. The incident occurred due to incorrect guidance provided by a remote Fleet Response specialist, which led to the self-driving vehicle getting stuck and requiring roadside assistance to complete its trip. This was described as a mistake made during the interaction, indicating an accidental nature of the failure [114925].
Duration temporary The software failure incident described in the article was temporary. The incident occurred when a Waymo self-driving taxi got confused by traffic cones, leading to it stopping abruptly and causing a partial traffic blockage. Remote assistance was called, but the vehicle continued to drive off before assistance could arrive. The incident involved incorrect guidance provided by a remote Fleet Response specialist, which made it challenging for the Waymo Driver to resume its intended route, requiring the intervention of Waymo's Roadside Assistance team to complete the trip [114925].
Behaviour crash, omission, other (a) crash: The software failure incident in the article can be categorized as a crash. The self-driving taxi experienced a situation where it stopped abruptly, drove off unexpectedly, and even attempted to escape when technicians arrived to help, indicating a loss of control and erratic behavior [114925]. (b) omission: The incident also involved an omission behavior where the self-driving van failed to perform its intended functions correctly. For example, it got confused by traffic cones, stopped abruptly, and blocked the road, omitting to navigate the situation smoothly [114925]. (c) timing: There is no specific indication in the article that the software failure incident was related to timing issues where the system performed its intended functions but at the wrong time. (d) value: The software failure incident did not involve the system performing its intended functions incorrectly in terms of providing incorrect outputs or results. (e) byzantine: The incident did not exhibit byzantine behavior where the system behaved inconsistently with varying responses and interactions. (f) other: The other behavior observed in this software failure incident could be described as erratic or unpredictable behavior. The self-driving taxi's actions of stopping abruptly, driving off unexpectedly, and attempting to escape can be considered as unusual and not falling into the typical categories of failure behaviors [114925].

IoT System Layer

Layer Option Rationale
Perception unknown (a) sensor: The software failure incident involving the Waymo self-driving taxi was not directly related to a sensor error. The incident was caused by incorrect guidance provided by a remote Fleet Response specialist, which made it challenging for the Waymo Driver to resume its intended route [114925]. (b) actuator: The failure was not directly related to an actuator error. The incident was primarily attributed to incorrect guidance given by the Fleet Response team, which led to difficulties for the Waymo Driver to continue its route [114925]. (c) processing_unit: The software failure incident was not directly linked to a processing error. The issue stemmed from the incorrect guidance provided by the remote Fleet Response specialist, which impacted the ability of the Waymo Driver to proceed with its intended route [114925]. (d) network_communication: The failure was not directly associated with a network communication error. The incident was primarily caused by the incorrect guidance given by the Fleet Response team, which affected the Waymo Driver's ability to navigate its route [114925]. (e) embedded_software: The software failure incident was indirectly related to embedded software error. The mistake made by the remote Fleet Response specialist led to challenges for the Waymo Driver to continue its intended route, requiring intervention from Waymo's Roadside Assistance team to complete the trip [114925].
Communication unknown The software failure incident reported in Article 114925 was not directly related to the communication layer of the cyber-physical system. The incident involved a self-driving taxi operated by Waymo that experienced confusion and issues while navigating through traffic cones and a closed lane, ultimately requiring roadside assistance to resolve the situation. The failure was attributed to a mistake made by a remote Fleet Response specialist, which led to incorrect guidance being provided to the self-driving vehicle, making it challenging for the vehicle to resume its intended route. This incident did not involve failures at the link_level or connectivity_level of the communication layer of the cyber-physical system [114925].
Application FALSE The software failure incident involving the Waymo self-driving taxi, as reported in Article 114925, was related to the application layer of the cyber physical system. The incident was caused by a mistake made by a remote Fleet Response specialist who provided incorrect guidance, making it challenging for the Waymo Driver to resume its intended route. This error introduced contributing factors that led to the failure, requiring Waymo's Roadside Assistance team to intervene and complete the trip [114925].

Other Details

Category Option Rationale
Consequence death, harm (a) death: People lost their lives due to the software failure - In September 2020, an Arizona woman was killed when she was hit by a self-driving Uber SUV whose safety driver was distracted watching an episode of the television show The Voice [Article 114925].
Domain transportation (a) The failed system was intended to support the transportation industry. The incident involved a self-driving taxi operated by Waymo in Chandler, Arizona, which experienced a software failure leading to confusion and erratic behavior on the road, requiring roadside assistance to intervene [Article 114925].

Sources

Back to List