Incident: Iowa Democratic Caucus 2020: App Failure Causes Reporting Chaos

Published Date: 2020-02-03

Postmortem Analysis
Timeline 1. The software failure incident happened on February 3, 2020 [95751, 95554, 95587, 95593].
System 1. The mobile app used to report figures to the state party during the Iowa caucuses failed [95751, 95554, 95587]. 2. The back-end results reporting system associated with the app also failed, leading to chaos in reporting results [95593]. 3. The quality-control system within the app failed to catch issues that resulted in jumbled results on caucus night [95593].
Responsible Organization 1. The Iowa Democratic Party, along with the company Shadow Inc., was responsible for causing the software failure incident [95593]. 2. The Democratic National Committee (DNC) was involved in the development and oversight of the app, which contributed to the failure incident [95629].
Impacted Organization 1. Iowa Democratic Party [95747, 95593] 2. Democratic National Committee [95629, 95593]
Software Causes 1. The delay in reporting results during the Iowa caucuses was primarily due to problems with a mobile app developed by Shadow Inc., which was used to collect and report voting tallies [95587]. 2. The app used for reporting results was not properly tested at a statewide scale, was quickly put together in just two months, and was not regarded as critical by developers in the final days before the caucuses [95593]. 3. The app experienced freezing issues, leading to precinct leaders resorting to calling in results, which overwhelmed the state party hotline and caused delays in reporting results [95593]. 4. The app was developed by Shadow Inc., a company recommended by Democratic leaders in Nevada, and the app's development timeline was rushed, with the contract for the app being finalized in the fall of 2019 [95593]. 5. The app worked well during testing, but no evidence of a bug that would cause result jumbling on caucus night was found, leading to issues when the app froze during actual use [95593].
Non-software Causes 1. Lack of proper testing at a statewide scale and rushed timeline for the app development [95587] 2. Issues with communication, training, and oversight by party officials [95593] 3. Problems with the reporting process, including delays, lack of preparation, and breakdown in communication [95593]
Impacts 1. The delay in reporting results from the Iowa caucuses caused chaos and uncertainty among Democratic operatives, volunteers, and caucus-goers, raising concerns about the technology underpinning American democracy [95751]. 2. The failure of the app led to a situation where precinct chairs struggled to transmit results, leading to a breakdown in the reporting process and a lack of transparency about the app's development and functionality [95751]. 3. The failure of the app resulted in precinct chairs facing technical errors, difficulties in using the app, and long hold times on the phone hotline, ultimately leading to a significant delay in reporting results [95554]. 4. The software failure incident highlighted the risks of introducing technology into voting processes, with experts emphasizing the need for thorough testing, security reviews, and caution when implementing technology in elections [95554]. 5. The failure of the app in the Iowa caucuses led to a situation where volunteers had to resort to manual methods of reporting results, such as passing around an iPad and emailing photos of handwritten forms, causing further delays and confusion [95593].
Preventions 1. Thorough Testing: Proper testing of the software at a statewide scale could have potentially identified issues and prevented the failure incident [95593]. 2. Early Planning and Development: Initiating the development of the app earlier and allowing more time for planning and development could have ensured a more robust and reliable system [95593]. 3. Transparent Communication: Improved communication between the state party officials, volunteers, and precinct leaders regarding the app's functionality, training, and reporting process could have helped in addressing issues more effectively [95587]. 4. Enhanced Oversight: Stronger oversight by party officials, including better training for volunteers, monitoring of the app's performance, and contingency planning, could have mitigated the impact of the failure incident [95593]. 5. Collaboration and Coordination: Improved coordination between the state party, national committee, and the technology vendor, along with clear roles and responsibilities, could have contributed to a smoother caucus process [95629]. 6. Security and Standards: Implementing robust security measures, adhering to standards for technology development, and involving cybersecurity experts in the process could have enhanced the app's reliability and integrity [95587].
Fixes 1. Proper testing of the software at a statewide scale before implementation [95587] 2. Thorough vetting and evaluation of the app by cybersecurity experts before public introduction [95587] 3. Increased scrutiny and oversight of caucus operations by the Democratic National Committee [95593] 4. Implementation of lessons learned from the Iowa caucus failure to prevent similar incidents in the future [95593]
References 1. The Washington Post [95751] 2. WIRED [95554] 3. CNN [95629, 95747, 95587, 95593]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: - The software failure incident in Iowa was related to an app created by Shadow Inc., a company recommended by Democratic leaders in Nevada as well. Shadow's CEO, Gerard Niemira, was a veteran of the 2016 Hillary Clinton campaign where he oversaw tech products for caucuses [95593]. - Shadow Inc. was involved in the Iowa caucus failure, and the company was also used by the Nevada Democratic Party for similar purposes [95587]. (b) The software failure incident having happened again at multiple_organization: - The software failure incident in Iowa was not properly tested at a statewide scale and was quickly put together in just the past two months [95587]. - The Iowa Democratic Party used an app developed by Shadow Inc., a for-profit technology company, which was also used by the Nevada Democratic Party [95587]. - The Iowa Democratic Party had issues with the app developed by Shadow Inc., leading to delays and chaos in reporting caucus results [95587]. - The Iowa caucus failure involved a mobile app developed by Shadow Inc., which was largely successful in its operation, but the failure was due to other factors such as intervention by the Democratic National Committee [108466].
Phase (Design/Operation) design, operation (a) In the Iowa caucus software failure incident, the failure due to the design phase was evident. The app used for reporting results was largely untested and proved difficult to use when needed the most. The software was developed by a company called Shadow Inc., recommended by Democratic leaders in Nevada, and the development timeline was rushed, with the app being put together in just two months before the caucuses [95587]. (b) The failure due to the operation phase was also significant in the Iowa caucus software failure incident. On the night of the caucuses, the app sometimes froze, and calls to the state party hotline were on hold for up to five hours, leading to chaos and delays in reporting results. The party had to resort to manual data entry and calculations due to the app's malfunction, causing errors in the reported data [95593].
Boundary (Internal/External) within_system, outside_system (a) within_system: - The failure of the app used in the Iowa caucuses was primarily due to issues within the system itself. The app, developed by Shadow Inc., had problems such as freezing, crashing, and not functioning properly when precinct leaders tried to report results [Article 95593]. - The app was rushed into development in just two months, leading to a lack of proper testing and training for precinct leaders on how to use it effectively [Article 95593]. - The app had coding issues that caused it to produce only partial and unreliable results, leading to a breakdown in the reporting process during the caucuses [Article 95593]. - The app was largely untested at a statewide scale, and there were concerns about its functionality, especially in areas with poor connectivity or high bandwidth use [Article 95587]. (b) outside_system: - The Democratic National Committee (DNC) was involved in ensuring the security of the app used in the Iowa caucuses. The DNC required the app makers to work with them on security measures, including providing access to system configurations, security controls, and operational plans [Article 95629]. - The DNC had requested changes to the app, such as a conversion tool for real-time access to reporting data, which led to delays and issues in the reporting process [Article 108466]. - The DNC had concerns about the security of the app and had proposed language for vendor contracts to ensure access to source code and testing of apps, but the Iowa party officials rejected these proposals [Article 95593].
Nature (Human/Non-human) non-human_actions (a) The software failure incident occurring due to non-human actions: - The delay in reporting results from the Iowa caucuses was partly due to problems with a mobile app used to report figures to the state party. The app was largely untested and proved difficult to use when needed most [95751]. - The app used for reporting results in the Iowa caucuses was not properly tested at a statewide scale and was quickly put together in just the past two months. It was developed by a company called Shadow Inc. and had issues such as freezing and crashing, leading to delays in reporting results [95587]. - The software failure in the Iowa caucuses was a total system breakdown that resulted from cascading failures, including
Dimension (Hardware/Software) software (a) The software failure incident occurring due to hardware: - There is no specific mention of the software failure incident in the articles being attributed to hardware issues. Therefore, there is no information available to support this option. (b) The software failure incident occurring due to software: - The software failure incident in the Iowa caucuses was primarily attributed to problems with the mobile app developed by Shadow Inc. used to report results to the state party. The app experienced issues such as freezing, crashing, and being difficult to use, leading to delays and errors in reporting results [95751, 95554, 95587, 95593].
Objective (Malicious/Non-malicious) non-malicious (a) The articles do not provide any information indicating that the software failure incident was malicious in nature. No evidence or mention of intentional harm or malicious intent by individuals is presented in the articles. (b) The software failure incident in Iowa's caucuses was non-malicious and stemmed from various contributing factors introduced without intent to harm the system. The failure was attributed to problems with a mobile app used to report results, lack of testing at a statewide scale, rushed development timeline, lapses in planning, failed oversight by party officials, poor training, and breakdown in communication between leaders and volunteers [95587]. The app developed by Shadow Inc. was largely untested at a statewide scale and was quickly put together in just two months [95593]. The app experienced freezing issues, and calls to the state party hotline were on hold for hours, leading to a chaotic reporting process [95593]. The failure was a total system breakdown that raised doubts about the management of the critical contest [95593]. The Iowa Democratic Party and the DNC were involved in the development and oversight of the app, with the DNC requesting security measures and involvement in the process [95629]. The app was not properly tested, and precinct leaders faced difficulties with the app, leading to a reliance on manual reporting methods [95593]. The app was used after another proposal for reporting votes was abandoned, and concerns were raised about its functionality and secrecy [95587]. The app was developed on a rushed timeline, with final tests conducted just days before the caucuses [95593]. The failure resulted in a breakdown in communication, reliance on manual data entry, and errors in reporting results [95593].
Intent (Poor/Accidental Decisions) poor_decisions (a) poor_decisions: Failure due to contributing factors introduced by poor decisions From Article 95593, the software failure incident in the Iowa caucuses was a result of cascading failures going back months, including
Capability (Incompetence/Accidental) development_incompetence (a) development_incompetence: The failure of the Iowa caucus reporting app was attributed to various factors related to development incompetence. The app was largely untested, difficult to use, and had not been used in any real election or tested at a statewide scale before the caucuses [95751]. The app was quickly put together in just two months, compressed on an already tight timeline, and the developers did not regard some critical updates as necessary [95593]. There were issues with the app freezing, precinct leaders struggling to report results, and calls to the state party hotline languishing on hold for hours, indicating a lack of proper testing and preparation [95587]. (b) accidental: The failure of the Iowa caucus reporting app was not attributed to accidental factors but rather to a series of cascading failures, lapses in planning, failed oversight, poor training, and a breakdown in communication between party officials and volunteers [95593]. The app worked well during testing, but no evidence of a bug that would cause issues on caucus night was found, indicating that the failure was not accidental but rather due to systemic issues [95593].
Duration permanent, temporary (a) The software failure incident in the Iowa caucuses can be considered temporary. The failure was due to a combination of factors such as problems with the mobile app, issues with reporting results, lack of proper testing, delays in development and approval of the app, poor training, breakdown in communication, and a cascade of failures in the reporting process [Article 95593]. (b) The software failure incident can also be seen as permanent to some extent. The issues in the Iowa caucuses were not just limited to the failure of the app but also included broader systemic failures like lapses in planning, failed oversight by party officials, poor training, and breakdown in communication between party leaders and volunteers [Article 95593].
Behaviour crash, omission, timing, value, other (a) crash: The software failure incident in the Iowa caucuses can be categorized as a crash. The app used to report results experienced technical glitches, freezing, and locking up, leading to delays and ultimately failing to perform its intended function of transmitting results effectively. This resulted in a chaotic situation where precinct leaders had to resort to calling in results, causing a breakdown in the reporting process [95587]. (b) omission: The software failure incident can also be categorized as an omission. The app omitted to perform its intended functions at instances where it froze, stalled, or locked up, leading to the need for precinct leaders to resort to manual reporting methods, such as calling in results. This omission of proper functioning contributed to the breakdown in the reporting process [95593]. (c) timing: The software failure incident can be categorized as a timing issue. The app performed its intended functions, but it did so too late, causing significant delays in the reporting of results. The delays in transmitting results were a key aspect of the failure incident, impacting the overall efficiency and accuracy of the caucus process [95747]. (d) value: The software failure incident can be categorized as a value issue. The app performed its intended functions incorrectly, leading to partial and unreliable results being reported. This incorrect performance of the app resulted in inconsistencies in the reported data, affecting the accuracy and reliability of the caucus results [95593]. (e) byzantine: The software failure incident does not align with the byzantine behavior category, which involves erroneous and inconsistent responses and interactions. The incident in the Iowa caucuses was more focused on technical glitches, delays, and errors in reporting rather than exhibiting complex and inconsistent behaviors [95587, 95593]. (f) other: The software failure incident can be categorized as a failure due to lack of testing at a statewide scale, rushed development, poor training, and breakdown in communication. These factors contributed to the overall breakdown in the reporting process, indicating a combination of technical issues, planning failures, and communication breakdowns that led to the software failure incident [95587, 95593].

IoT System Layer

Layer Option Rationale
Perception processing_unit, network_communication, embedded_software (a) sensor: Failure due to contributing factors introduced by sensor error - The failure in reporting the Iowa caucus results was not directly related to a sensor error as there was no mention of sensor technology being a contributing factor to the software failure incident in the articles. (b) actuator: Failure due to contributing factors introduced by actuator error - The failure in reporting the Iowa caucus results was not directly related to an actuator error as there was no mention of actuator technology being a contributing factor to the software failure incident in the articles. (c) processing_unit: Failure due to contributing factors introduced by processing error - The failure in reporting the Iowa caucus results was related to processing errors introduced by the software app used to report the results. The app experienced glitches, freezes, and delays in processing the data, leading to a breakdown in the reporting process [Article 95587]. (d) network_communication: Failure due to contributing factors introduced by network communication error - The failure in reporting the Iowa caucus results was partially related to network communication errors. The app used for reporting results froze, and calls to the state party hotline were on hold for hours, indicating issues with network communication [Article 95593]. (e) embedded_software: Failure due to contributing factors introduced by embedded software error - The failure in reporting the Iowa caucus results was directly related to embedded software errors. The app used for reporting results, developed by Shadow Inc., experienced glitches, freezes, and failures, leading to delays and inaccuracies in reporting the caucus results [Article 95587].
Communication connectivity_level (a) The failure was not related to the communication layer of the cyber physical system that failed. There is no specific mention of the failure being due to contributing factors introduced by the wired or wireless physical layer in the articles. (b) The failure was related to the connectivity level. The failure of the app used in the Iowa Democratic caucuses was due to issues with the mobile app used to report results. The app developed by Shadow Inc. faced problems such as freezing, crashing, and not functioning properly, leading to delays in reporting results [95593]. Additionally, there were issues with the hotline being overwhelmed, causing delays in reporting results [95554]. The app was largely untested, developed in a rushed timeline, and faced usability issues, contributing to the failure at the network or transport layer of the cyber physical system.
Application TRUE The software failure incident related to the application layer of the cyber physical system that failed due to bugs, operating system errors, unhandled exceptions, and incorrect usage is evident in the Iowa Democratic caucuses. The failure was primarily attributed to the mobile app developed by Shadow Inc., which was used to report and tabulate caucus results. The app was rushed into development in just two months, largely untested at a statewide scale, and faced issues such as freezing and crashing during caucus night [Article 95587]. Additionally, the app required precinct leaders to log back in with a PIN if it stalled, froze, or locked up, leading to confusion and delays in reporting results. The app's failure led to precinct leaders resorting to calling in results, which overwhelmed the state party hotline and caused significant delays in reporting results [Article 95593].

Other Details

Category Option Rationale
Consequence property, delay, non-human (a) death, (b) harm, (c) basic, (d) property, (e) delay, (f) non-human, (g) no_consequence, (h) theoretical_consequence, (i) other The software failure incident in the Iowa caucuses did not result in any deaths or physical harm to individuals. There were no direct consequences related to access to basic needs like food or shelter. However, the incident did lead to delays in reporting the caucus results, causing significant chaos and uncertainty [95554]. The property in terms of material goods, money, or data was impacted as the software failure affected the reporting of election results, leading to a breakdown in the caucus process [95593]. The non-human entities, in this case, the software app developed by Shadow Inc., were directly impacted by the failure, resulting in the inability to accurately report and tabulate the caucus results [95587]. There were no theoretical consequences discussed in the articles, but the incident did have real-world implications in terms of the integrity of the election process.
Domain information, government (a) The failed system was intended to support the information industry. The system was a smartphone app used to report results from the Iowa Democratic caucuses [Article 95751]. (b) The failed system was not related to the transportation industry. (c) The failed system was not related to the natural resources industry. (d) The failed system was not related to the sales industry. (e) The failed system was not related to the construction industry. (f) The failed system was not related to the manufacturing industry. (g) The failed system was not related to the utilities industry. (h) The failed system was not related to the finance industry. (i) The failed system was not related to the knowledge industry. (j) The failed system was not related to the health industry. (k) The failed system was not related to the entertainment industry. (l) The failed system was intended to support the government industry. It was used for reporting caucus results in the Iowa Democratic caucuses [Article 95593]. (m) The failed system was not related to an industry not described in the options.

Sources

Back to List