Incident: Unmanned Drone Crash Due to Software Override in Poor Weather

Published Date: 2016-12-27

Postmortem Analysis
Timeline 1. The software failure incident involving the army drone crashing after the pilot switched off safety systems happened on November 2, 2015, as reported in Article 57781.
System 1. Automatic landing system 2. Manual override button 3. Safety protections 4. Master Override (MO) system
Responsible Organization 1. The pilot who switched off safety systems causing the drone to crash [57781] 2. Lack of flying expertise and organization within the project team involved in the Watchkeeper drone project [57781]
Impacted Organization 1. The Army 2. The Defence Accident Investigation Branch of the Defence Safety Authority 3. The crew involved in the training mission 4. The developers of the aircraft 5. The overall program governance and supervision 6. The aviation community 7. The public perception of the aircraft's capabilities [Cited from Article 57781]
Software Causes 1. The software caused the drone to self-abort its approach to landing in poor weather conditions, such as low-cloud, fog, gusts, precipitation, or a combination of these factors, due to anomalies in the system [57781]. 2. The manual override button, when pressed by the pilots, turned off many safety protections of the aircraft's software, leading to confusion in flying through low-level fog and ultimately causing the crash [57781]. 3. The software's automatic landing system worked as designed in good weather but had issues in poor weather conditions, leading to the need for manual intervention which increased the risk of the air vehicle crashing [57781].
Non-software Causes 1. Lack of flying expertise and organization among those involved in the project [57781] 2. Premature selection of manual override by the crew despite having plenty of fuel and the option to wait for weather conditions to improve [57781] 3. Misleading claims in the official documentation about the drone being an all-weather aircraft, leading to unrealistic expectations [57781] 4. Lack of appropriate guidance and warnings in the official documents regarding limitations or cautions concerning cloud or low visibility on landing [57781] 5. Disappointing organizational, control of activity, and governance issues highlighted by the investigation [57781]
Impacts 1. The software failure incident led to the crash and destruction of a £1 million army drone during a training mission at Boscombe Down [57781]. 2. The incident highlighted the limitations of the drone in poor visibility conditions such as fog, rain, and snow, causing it to become easily confused and ultimately crash [57781]. 3. The lack of flying expertise and organization within the project, as criticized in the report, indicated a negative impact on the overall operation and safety of the drone system [57781]. 4. The incident raised concerns about the need for software fixes and improvements before the drone could provide a reliable and credible capability in various weather conditions [57781]. 5. The report emphasized the disappointment in organizational, control of activity, and governance issues, indicating broader impacts beyond just the technical aspects of the software failure incident [57781].
Preventions 1. Proper training and expertise for the pilots involved in controlling the drone could have prevented the software failure incident [57781]. 2. Improved organization and supervision during the training mission could have helped prevent the incident [57781]. 3. Clearer documentation and warnings about the limitations and cautions concerning cloud or low visibility on landing in the official documents could have prevented the incident [57781]. 4. Fixing the software issues identified in the investigation report before operating the drone in a range of weather conditions could have prevented the incident [57781].
Fixes 1. Fixing the software to address anomalies in poor weather conditions, such as low-cloud, fog, gusts, and precipitation, that can cause the automatic landing system to self-abort its approach to landing [57781]. 2. Providing appropriate guidance and warnings in the official documentation about the limitations and cautions concerning cloud or low visibility on landing to align operators' expectations with the true capabilities of the aircraft [57781]. 3. Enhancing the supervision, control of activity, and governance to ensure proper decision-making during training missions and operations [57781]. 4. Implementing a framework that emphasizes aviation-mindedness and the application of judgement and airmanship by experienced aviators to ensure safe and reliable operation of UAVs [57781].
References 1. The investigation report into the army drone crash incident at Boscombe Down [57781]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident has happened again at one_organization: - The article mentions that weeks earlier, another high-tech surveillance aircraft crashed while attempting to land in Wales in poor weather conditions [57781]. This indicates a similar incident occurred with another aircraft within the same organization or project. (b) The software failure incident has happened again at multiple_organization: - There is no specific mention in the article about the software failure incident happening at multiple organizations.
Phase (Design/Operation) design, operation (a) The software failure incident in the article can be attributed to design-related factors introduced during the system development phase. The report highlighted issues with the automatic landing system of the drone, stating that it works as designed in good weather but can encounter anomalies in poor weather conditions like low-cloud, fog, gusts, or precipitation. These anomalies can lead to the air vehicle self-aborting its approach to landing. The selection of the Master Override (MO) can remove the in-built safety protection, increasing the risk of a crash. The report emphasized that the software needed to be fixed before the drone could provide a reliable capability in various weather conditions, indicating design flaws in the system [57781]. (b) Additionally, the incident also involved operational factors related to the misuse of the system. The crew operating the drone was criticized for prematurely selecting the manual override button, which turned off many safety protections, leading to the crash. The report mentioned that the crew could have waited for weather conditions to improve instead of forcing the vehicle to land by using the manual override. This operational decision contributed to the software failure incident [57781].
Boundary (Internal/External) within_system, outside_system (a) The software failure incident involving the army drone crashing was primarily within the system. The incident was caused by the pilot pressing the manual override button, which turned off many safety protections of the aircraft's automatic landing system. This action led to the drone crashing on the runway at Boscombe Down [57781]. The report highlighted that the decision to select the Master Override (MO) was significant as it increased the risk of the air vehicle crashing. It also mentioned that the software needed to be fixed before the drone could provide a reliable and credible capability in a range of weather conditions [57781]. (b) However, there were also contributing factors outside the system that affected the incident. The report criticized the lack of flying expertise and organization within the project, indicating issues with supervision during the training mission and disappointing organizational control of activity and governance [57781]. Additionally, the report mentioned that the official documentation of the aircraft claimed it was an all-weather aircraft, which was not true, leading to operators having expectations beyond its true capabilities. This lack of accurate information in the documentation could be considered an external factor influencing the incident [57781].
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident in the article was primarily due to non-human actions. The incident occurred because the automatic landing system of the drone had anomalies in poor weather conditions such as low-cloud, fog, gusts, or precipitation. These anomalies could cause the air vehicle to self-abort its approach to landing. For example, false height readings from the laser altimeters due to fog or low cloud below the vehicle could prompt the system to self-abort the landing approach. Additionally, wind gusts or sudden vehicle maneuvers could also trigger the system to self-abort the landing attempt [57781]. (b) Human actions also played a significant role in the software failure incident. The pilots pressed the manual override button, which turned off many safety protections of the drone. This action was taken when the aircraft was still flying at 22 feet, and the sensors believed it was on the ground, causing the drone to plunge out of the sky at a 35-degree nose-down angle. The report criticized the crew for being premature in selecting the manual override as the aircraft still had plenty of fuel, and they could have waited for weather conditions to improve. The decision to select the Master Override (MO) was highlighted as significant as it increased the risk of the air vehicle crashing [57781].
Dimension (Hardware/Software) hardware, software (a) The software failure incident in the article was primarily due to hardware-related factors. The incident involved a drone crash after the pilot pressed the manual override button, which turned off many safety protections, leading to confusion in the system caused by a false reading from the laser altimeter. This hardware-related issue contributed to the crash of the drone [57781]. (b) The software failure incident also had contributing factors originating in software. The automatic landing system of the drone was reported to work as designed in good weather conditions but faced anomalies in poor weather conditions, such as low-cloud, fog, gusts, or precipitation. The software's inability to handle these conditions led to the self-abort of the landing approach. Additionally, the decision to select the Master Override (MO) in the software increased the risk of the air vehicle crashing. The report highlighted that the software needed to be fixed before the drone could provide a reliable capability in various weather conditions [57781].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident described in the articles is non-malicious. The incident was primarily attributed to human error and lack of expertise rather than any malicious intent. The failure occurred due to the pilot's decision to press the manual override button, which turned off safety protections and led to the drone crashing during landing attempts in poor weather conditions [57781]. The investigation report highlighted issues with the software's response to adverse weather conditions and the lack of proper guidance in the official documentation, indicating a non-malicious failure scenario.
Intent (Poor/Accidental Decisions) poor_decisions, accidental_decisions (a) The software failure incident was related to poor decisions made by the pilots. The incident occurred when the pilots pressed the manual override button, which turned off many safety protections, causing the drone to crash. The report criticized the crew for being 'premature' in selecting the manual override as the aircraft still had plenty of fuel, and they could have waited for weather conditions to improve [57781]. (b) The software failure incident was also related to accidental decisions or unintended consequences. The automatic landing system of the drone was designed to work in good weather conditions, but anomalies in the system could cause the air vehicle to self-abort its approach to landing in poor weather conditions like low-cloud, fog, gusts, or precipitation. The selection of the Master Override (MO) to force the vehicle to land removed the in-built safety protection, increasing the risk of a crash. The report highlighted that the decision to select MO was significant and could lead to the air vehicle crashing [57781].
Capability (Incompetence/Accidental) development_incompetence (a) The software failure incident in the article was related to development incompetence. The investigation report criticized the lack of flying expertise and organization within the £1.2 billion project, highlighting issues with supervision during the training mission and the need for fixing the software before the drone could provide a reliable capability in various weather conditions. The report also mentioned that the software needed to be fixed and that technical issues would be addressed [57781]. (b) The incident was not attributed to accidental factors but rather to the lack of professional competence and oversight in the development and operation of the drone software.
Duration temporary The software failure incident described in the articles can be categorized as a temporary failure. The incident occurred when the pilots pressed the manual override button, which turned off many safety protections, causing the drone to crash. The report highlighted that the decision to select the Master Override (MO) was significant as it increased the risk of the air vehicle crashing [57781]. Additionally, the investigation concluded that the crew was premature in selecting the manual override as the aircraft still had plenty of fuel, and they could have waited for weather conditions to improve [57781]. This indicates that the failure was temporary and could have been avoided or mitigated by different actions taken by the operators.
Behaviour crash, omission, other (a) crash: The software failure incident in this case led to a crash of the army drone. The drone crashed on the runway at Boscombe Down after the pilot pressed the manual override button, which closed down several safety systems, causing the drone to plunge out of the sky at a 35-degree nose-down angle [57781]. (b) omission: The software failure incident also involved an omission where the system failed to perform its intended functions. The automatic landing system of the drone was designed to work in good weather conditions but in poor weather conditions like low-cloud, fog, gusts, or precipitation, anomalies in the system could make the air vehicle self-abort its approach to landing. The system could register false height due to fog or low cloud below the vehicle, leading to the drone aborting its landing approach [57781]. (c) timing: The software failure incident did not specifically involve a timing issue where the system performed its intended functions correctly but too late or too early. (d) value: The software failure incident did not involve the system performing its intended functions incorrectly. (e) byzantine: The software failure incident did not involve the system behaving erroneously with inconsistent responses and interactions. (f) other: The software failure incident involved a situation where the pilots had the option to select the Master Override (MO) which would remove the in-built safety protection of the automatic landing system, causing the drone to continue its approach unless a manual abort was commanded by the crew. This decision to select MO was significant as it increased the risk of the air vehicle crashing. The report criticized the crew for being 'premature' in selecting the manual override when the aircraft still had plenty of fuel, and they could have waited for weather conditions to improve [57781].

IoT System Layer

Layer Option Rationale
Perception sensor (a) sensor: The software failure incident was related to sensor error. The article mentions that the system got confused flying through low-level fog after the laser altimeter registered a false reading, prompting the crash which wrote-off the £1 million aircraft [57781].
Communication unknown The software failure incident described in the article does not directly point to a failure related to the communication layer of the cyber-physical system. Instead, the incident primarily highlights issues related to the automatic landing system, manual override functionality, weather conditions affecting sensors, lack of flying expertise, and organizational shortcomings. The failure was more attributed to the interaction between the software controlling the drone and the environmental conditions, pilot decisions, and system limitations rather than a specific communication layer failure within the cyber-physical system.
Application FALSE The software failure incident described in the provided article [57781] was not directly related to the application layer of the cyber physical system. The incident was primarily attributed to the pilot's decision to override safety systems, leading to confusion in poor weather conditions and ultimately causing the drone to crash. The failure was more related to operational and decision-making aspects rather than bugs, operating system errors, unhandled exceptions, or incorrect usage typically associated with application layer failures.

Other Details

Category Option Rationale
Consequence property, non-human, theoretical_consequence (a) death: People lost their lives due to the software failure - The incident involving the army drone crashing due to the pilot pressing the manual override button resulted in the destruction of the £1 million aircraft [57781]. (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur - The report mentioned that the decision to select the Master Override (MO) in poor weather conditions significantly increased the risk of the air vehicle crashing, but it was noted that the crew could have waited for weather conditions to improve instead of prematurely selecting the manual override [57781].
Domain transportation, government (a) The failed system was intended to support the defense industry. The incident involved an army drone, specifically the Watchkeeper drone, which crashed during a training mission at Boscombe Down [Article 57781]. The drone was part of a £1.2 billion project and was being controlled remotely by pilots using radio signals. The investigation report highlighted issues with flying expertise and organization within the defense sector related to the project [Article 57781]. (l) The failed system was also related to the government sector. The incident involved a high-tech surveillance aircraft that crashed during an attempt to land in poor weather conditions while returning from a training mission over Sailsbury Plain range [Article 57781]. The report by the Defence Accident Investigation Branch of the Defence Safety Authority criticized the lack of supervision during the training mission and highlighted disappointing organizational and governance issues within the government sector [Article 57781].

Sources

Back to List