Incident: Self-Driving Car AI Software Failure at Roborace UK Event

Published Date: 2020-10-30

Postmortem Analysis
Timeline 1. The software failure incident of the self-driving car crashing into a wall during the Roborace in the UK happened in October 2020 as per the article published on October 30, 2020 [106708].
System 1. Acronis SIT Autonomous software system [106708]
Responsible Organization 1. The software failure incident in the Roborace event, where the Acronis SIT Autonomous vehicle crashed into a wall, was caused by the AI system operating the vehicle [106708].
Impacted Organization 1. Acronis SIT Autonomous team and their vehicle were impacted by the software failure incident [106708].
Software Causes 1. The software cause of the failure incident in the Roborace where a self-driving car crashed into a wall was likely a programming error or bug in the AI system that controlled the vehicle's acceleration and steering, leading to the unexpected turn and collision [106708].
Non-software Causes 1. Accelerator malfunction leading to unexpected turn into the wall [106708] 2. Physical collision with the pit-lane barrier [106708]
Impacts 1. The software failure incident led to the self-driving car operated by the Acronis SIT Autonomous team hitting a pit-lane barrier during a Roborace event in the UK, causing significant damage to the vehicle's front-end components such as the AI camera, LIDAR system, and sensors [106708]. 2. The incident resulted in the team's car being out of the race before it even started, showcasing the challenges and hurdles autonomous vehicles still face in terms of reliability and safety [106708]. 3. The crash highlighted the potential risks associated with autonomous driving technology, especially as other companies like Tesla and Waymo are also working on advancing autonomous vehicle capabilities [106708].
Preventions 1. Implementing more robust testing procedures to ensure the AI system's behavior in various scenarios, including unexpected accelerations and turns, could have prevented the software failure incident [106708]. 2. Conducting thorough simulations and real-world testing to identify and address any potential issues or malfunctions in the software controlling the self-driving car could have helped prevent the incident [106708]. 3. Enhancing the fail-safe mechanisms within the software to quickly detect and respond to anomalies or errors in the AI's decision-making process could have mitigated the risk of such accidents [106708].
Fixes 1. Implement more robust testing procedures to ensure the software controlling the self-driving car is thoroughly tested for various scenarios and edge cases to prevent unexpected behavior like accelerating into a wall [106708]. 2. Conduct a thorough review of the software code and algorithms to identify any potential bugs or issues that could lead to such incidents and make necessary corrections [106708]. 3. Enhance the fail-safe mechanisms in the software to quickly detect anomalies and take corrective actions to prevent accidents, such as emergency braking or steering corrections [106708].
References 1. Car and Driver [106708] 2. Team Principal of Acronis SIT Autonomous Team [106708]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident having happened again at one_organization: The incident involving the Acronis SIT Autonomous self-driving car crashing into a wall during the Roborace in the UK can be considered a software failure within the same organization. This incident showcases a failure in the software controlling the vehicle, leading to a crash before the race even began [106708]. (b) The software failure incident having happened again at multiple_organization: There is no specific mention in the provided article about similar incidents happening at other organizations or with their products and services. Therefore, it is unknown if similar software failure incidents have occurred at multiple organizations based on the information available.
Phase (Design/Operation) design, operation (a) The software failure incident in the article can be attributed to the design phase. The incident occurred during a Roborace event in the UK where a self-driving car, the Acronis SIT Autonomous, unexpectedly accelerated and crashed into a wall before even starting the race. The article mentions that the vehicle hit the pit-lane barrier due to the AI making an unexpected turn to the right, indicating a design flaw in the system's programming or decision-making algorithms [106708]. (b) The software failure incident can also be linked to the operation phase. The article describes how the Acronis SIT Autonomous vehicle, operated by a Swiss team, was gearing up for the race when the AI hit the accelerator and made the unexpected turn, leading to the crash. This indicates that the failure was a result of the operation or execution of the system during the event, possibly due to a miscommunication or misinterpretation of commands by the AI system [106708].
Boundary (Internal/External) within_system (a) The software failure incident in the article is primarily within_system. The incident occurred during a Roborace event in the UK where a self-driving car, the Acronis SIT Autonomous, unexpectedly accelerated and crashed into a wall before the race even started. The failure was attributed to the AI of the vehicle hitting the accelerator and making an unexpected turn, leading to the collision with the pit-lane barrier. The incident resulted in damage to the front-end components of the vehicle, including the AI camera, LIDAR system, and sensors responsible for determining the direction of travel [106708].
Nature (Human/Non-human) non-human_actions (a) The software failure incident in the Roborace involving the Acronis SIT Autonomous vehicle crashing into a wall was due to non-human actions. The incident occurred when the AI of the vehicle hit the accelerator and made an unexpected turn to the right, resulting in the vehicle hitting the pit-lane barrier [106708]. (b) The article does not provide information indicating that the software failure incident was due to contributing factors introduced by human actions.
Dimension (Hardware/Software) hardware, software (a) The software failure incident in the article was primarily due to hardware issues. The self-driving car operated by the Swiss team experienced a failure when the AI hit the accelerator and made an unexpected turn, resulting in the vehicle smashing into a pit-lane barrier and damaging the front-end components like the AI camera, LIDAR system, and sensors [106708]. (b) The software failure incident also had contributing factors originating in software. The article mentions that the Roborace showcased six cars built with the same technology but using different software. The specific software running on the Acronis SIT Autonomous vehicle likely played a role in the unexpected turn that led to the crash [106708].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident described in the articles does not appear to be malicious. It was a non-malicious failure where the self-driving car operated by the Swiss team, Acronis SIT Autonomous, experienced a glitch or error that caused it to accelerate unexpectedly and crash into a wall during the Roborace event in the UK [106708]. The incident was described as an unexpected turn to the right that led to the collision with the pit-lane barrier, resulting in damage to the front-end components of the vehicle such as the AI camera, LIDAR system, and sensors [106708]. The commentators at the event expressed surprise and disappointment at the outcome, indicating that the incident was not intentional but rather a result of a software or technical issue [106708].
Intent (Poor/Accidental Decisions) accidental_decisions (a) The intent of the software failure incident was not due to poor decisions but rather an accidental decision made by the AI in the self-driving car during the Roborace incident. The AI in the Acronis SIT Autonomous vehicle hit the accelerator and made an unexpected turn to the right, resulting in the car smashing into a wall [106708]. This accidental decision led to the failure during the race.
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident in the Roborace where the Acronis SIT Autonomous vehicle crashed into a wall can be attributed to development incompetence. The incident occurred as the AI hit the accelerator and made an unexpected turn to the right, hitting the pit-lane barrier [106708]. This failure showcases the challenges and limitations that autonomous vehicles still face, despite advancements in technology and AI solutions designed for accurate results. (b) The software failure incident can also be considered accidental as it was not intentional for the vehicle to crash into the wall. The unexpected turn taken by the AI, resulting in the collision, can be seen as an accidental failure during the race [106708].
Duration temporary (a) The software failure incident described in the articles seems to be temporary. The incident occurred during a Roborace event in the UK where a self-driving car, the Acronis SIT Autonomous, unexpectedly accelerated and crashed into a wall before the race even started. The failure was attributed to the AI hitting the accelerator and making an unexpected turn, leading to the collision [106708]. This incident was a one-time event during the race and not a permanent failure caused by all circumstances.
Behaviour crash, omission, other (a) crash: The software failure incident in the article resulted in a crash where the self-driving car operated by the Acronis SIT Autonomous team smashed into a wall during a Roborace event in the UK. The AI hit the accelerator unexpectedly, causing the vehicle to make a sharp right turn and collide with the pit-lane barrier, leading to a physical crash [106708]. (b) omission: The software failure incident can also be categorized as an omission failure as the self-driving car omitted to perform its intended function of navigating through the raceway without hitting any obstacles. Instead, the vehicle failed to avoid the real obstacle (pit-lane barrier) and crashed into it, indicating an omission of the expected behavior [106708]. (c) timing: There is no specific mention of a timing-related failure in the incident described in the article. (d) value: The software failure incident did not involve a value-related failure where the system performed its intended functions incorrectly. (e) byzantine: The software failure incident does not align with a byzantine failure where the system behaves erroneously with inconsistent responses and interactions. (f) other: The other behavior observed in this software failure incident is a deviation from the expected behavior during a competitive event. The self-driving car's unexpected turn and subsequent crash deviated from the planned course of action, leading to a disruptive and unexpected outcome in the race [106708].

IoT System Layer

Layer Option Rationale
Perception sensor, actuator, embedded_software (a) The failure was related to the sensor layer of the cyber physical system that failed: - The incident involved the self-driving car, Acronis SIT Autonomous, hitting a pit-lane barrier due to a wrong turn made by the AI, which relies on sensors for perception [106708]. (b) The failure was related to the actuator layer of the cyber physical system that failed: - The self-driving car accelerated unexpectedly and made a sharp right turn, leading to the collision with the wall, indicating a failure in the actuator system [106708]. (c) The failure was related to the processing unit layer of the cyber physical system that failed: - The incident did not specifically mention a failure related to the processing unit layer [106708]. (d) The failure was related to the network communication layer of the cyber physical system that failed: - The incident did not indicate any network communication issues as a contributing factor to the failure [106708]. (e) The failure was related to the embedded software layer of the cyber physical system that failed: - The article mentions that the self-driving car's software was different from other cars in the race, suggesting that the embedded software may have played a role in the unexpected turn and subsequent crash [106708].
Communication unknown The software failure incident described in the articles does not directly point to a failure related to the communication layer of the cyber physical system. The incident primarily involved the self-driving car, Acronis SIT Autonomous, making an unexpected turn and crashing into a wall during a Roborace event. The focus was on the AI system of the vehicle malfunctioning and causing the crash, rather than issues related to the communication layer of the cyber physical system. Therefore, the failure was not specifically attributed to the link_level or connectivity_level aspects of the system.
Application FALSE The software failure incident described in the articles does not seem to be related to the application layer of the cyber physical system. The incident involved a self-driving car in a Roborace event that crashed into a wall due to unexpected behavior of the AI, rather than being caused by bugs, operating system errors, unhandled exceptions, or incorrect usage typically associated with application layer failures. Therefore, the failure was not related to the application layer as defined in the question.

Other Details

Category Option Rationale
Consequence property, delay, non-human, theoretical_consequence (a) death: There were no reports of any deaths resulting from the software failure incident described in the articles [106708]. (b) harm: The article mentions that no one was physically hurt during the crash, only a few egos were affected [106708]. (c) basic: There is no mention of people's access to food or shelter being impacted by the software failure incident [106708]. (d) property: The software failure incident resulted in property damage as the self-driving car smashed into a wall, damaging the front-end components like the AI camera, LIDAR system, and sensors [106708]. (e) delay: The incident caused a delay in the race as one of the self-driving cars crashed into a wall before even starting the race [106708]. (f) non-human: The software failure incident impacted the self-driving car itself, resulting in damage to its components and hindering its participation in the race [106708]. (g) no_consequence: The software failure incident did have observable consequences, such as property damage and a delay in the race [106708]. (h) theoretical_consequence: There were discussions about the potential consequences of the incident in terms of the challenges autonomous vehicles still face, as highlighted by the crash, but these potential consequences did not materialize into actual harm or fatalities [106708]. (i) other: There were no other consequences mentioned in the articles beyond the property damage, delay in the race, and the theoretical discussions about the challenges faced by autonomous vehicles [106708].
Domain transportation, entertainment (a) The failed system was intended to support the entertainment industry. The incident involved a self-driving car participating in a Roborace event, which is a competition showcasing autonomous driving technology in a racing setting [106708].

Sources

Back to List