Incident: Inaccurate Best-Selling Book Lists Due to Data Merging Software Failure

Published Date: 2019-07-26

Postmortem Analysis
Timeline 1. The software failure incident happened in late May 2018 [87968].
System 1. Software that merges data from NPD BookScan and Amazon 2. Data ingestion and combination process within the software 3. Change in one data source that led to errors in the software The failures occurred in the software system responsible for merging data from NPD BookScan and Amazon, specifically in the data ingestion and combination process, and due to a change in one of the data sources. [87968]
Responsible Organization 1. The software failure incident was caused by issues related to how the software ingested and combined data from the two sources, NPD BookScan and Amazon [87968]. 2. Another cause of the software failure incident was a change in one data source that led the software to inadvertently remove the top bestseller, resulting in the wrong order of books and ignoring some sales from top-selling titles [87968].
Impacted Organization 1. The Washington Post [87968]
Software Causes 1. The software failure incident was caused by issues related to how the software ingested and combined data from the two sources [87968]. 2. Another cause of the failure was a change in one data source in late December 2018, which led the software to inadvertently remove the top bestseller, causing the lists to be inaccurate and ignoring some sales from top-selling titles [87968].
Non-software Causes 1. A change in one data source in late December 2018 led to the software inadvertently removing the top bestseller, causing the books to be listed in the wrong order and ignoring some sales from top-selling titles [87968].
Impacts 1. Inaccurate best-selling book lists were published, affecting the credibility and reliability of the rankings [87968]. 2. Sales data from top-selling titles were ignored, potentially leading to financial losses for authors and publishers [87968]. 3. The software failure incident required manual intervention for list verification before publication, increasing operational workload and potentially delaying the release of the lists [87968].
Preventions 1. Implementing thorough testing procedures before deploying software updates or changes could have potentially prevented the software failure incident [87968]. 2. Conducting regular audits and checks on data sources and their integration within the software could have helped identify issues earlier and prevent errors from occurring [87968]. 3. Implementing automated monitoring systems to detect anomalies or discrepancies in the merged data sets could have provided early warnings about potential issues with the software [87968].
Fixes 1. Implement thorough testing procedures to catch any data ingestion and merging errors before they impact the accuracy of the lists [87968]. 2. Regularly monitor and verify the data sources to ensure that any changes do not inadvertently affect the software's functionality [87968]. 3. Consider implementing automated checks or alerts to detect anomalies in the lists promptly [87968]. 4. Enhance the software's error handling mechanisms to prevent the inadvertent removal of top bestsellers or other critical data [87968].
References 1. NPD BookScan 2. Amazon 3. The Washington Post's software team 4. The Washington Post's publication team

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident related to inaccurate book lists due to errors in merging data from different sources happened again at The Washington Post. The incident occurred in late May 2018 and was not fully resolved until June 2019, indicating a recurrence within the same organization [87968]. (b) There is no information in the provided article to suggest that a similar incident has happened at other organizations or with their products and services.
Phase (Design/Operation) design (a) The software failure incident in the article was related to the design phase. The errors in merging data from two sources and the inadvertent removal of the top bestseller were attributed to how the software ingested and combined data from the sources, as well as a change in one data source that affected the software's functionality [87968].
Boundary (Internal/External) within_system (a) The software failure incident described in Article 87968 was primarily within the system. The errors were related to how the software ingested and combined data from different sources, as well as a change in one data source that led to unintended consequences within the software itself. These internal factors contributed to the inaccuracies in the lists of best-selling books. The article mentions that the errors were caught, corrected, and the process was improved by hand-checking every list before publication to ensure accuracy, indicating an internal focus on addressing the software issues [87968].
Nature (Human/Non-human) non-human_actions (a) The software failure incident in Article 87968 was primarily due to non-human_actions. The errors in the best-selling books lists were caused by issues related to how the software ingested and combined data from different sources, as well as a change in one data source that led to the software inadvertently removing the top bestseller and causing the books to be listed inaccurately. These issues were not directly caused by human actions but rather by changes in data sources and software behavior [87968].
Dimension (Hardware/Software) software (a) The software failure incident described in Article 87968 was primarily due to contributing factors originating in software. The incident involved errors in the software that merged data from different sources, leading to inaccuracies in the lists of best-selling books. Specifically, issues arose from how the software ingested and combined data from NPD BookScan and Amazon, as well as a change in one data source that caused the software to inadvertently remove the top bestseller and list books incorrectly. These software-related issues resulted in the need for manual checking and corrections before publication [87968].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident described in Article 87968 was non-malicious. The errors in the lists of best-selling books were attributed to issues in how the software ingested and combined data from different sources, as well as a change in one data source that inadvertently caused the software to remove the top bestseller and list books in the wrong order. These issues were not intentional but rather resulted from unintended consequences of software changes and data integration processes [87968].
Intent (Poor/Accidental Decisions) poor_decisions, accidental_decisions (a) The software failure incident related to the intent of poor_decisions can be seen in the article. The incident was caused by two identified factors. One factor was related to how the software ingested and combined data from the two sources, indicating a potential poor decision in the design or implementation of that process. The other factor started when a change in one data source led the software to inadvertently remove the top bestseller, causing the lists to be inaccurate, which could also be attributed to a poor decision in managing data sources [87968].
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident related to development incompetence is evident in the article as it mentions that the errors in the lists were caused by issues in how the software ingested and combined data from the two sources, as well as a change in one data source that led to the software inadvertently removing the top bestseller. These issues were not caught until June 2019, indicating a lack of professional competence in ensuring the accuracy of the software's data processing and handling [87968]. (b) The software failure incident also had accidental contributing factors, as mentioned in the article. The change in one data source that led to the software inadvertently removing the top bestseller was not intentional but occurred accidentally, causing the lists to be inaccurate and sales from top-selling titles to be ignored [87968].
Duration temporary The software failure incident described in Article 87968 was temporary. The incident started in late May 2018 and continued until June 2019 when the errors were caught and corrected. The errors were intermittent and were caused by issues in how the software ingested and combined data from different sources, as well as a change in one data source that led to further inaccuracies. The fact that the errors were eventually identified, corrected, and the process was improved by hand-checking every list before publication indicates that the failure was not permanent but rather temporary [87968].
Behaviour omission, value, other (a) crash: The software failure incident did not involve a crash where the system lost state and did not perform any of its intended functions. The issue was related to inaccurate data merging and incorrect listing of best-selling books, but the system did not completely stop functioning. (b) omission: The software failure incident involved omission where the system omitted to perform its intended functions by inadvertently removing the top bestseller, causing the books to be listed in the wrong order and ignoring some sales from top-selling titles [87968]. (c) timing: The software failure incident did not involve timing issues where the system performed its intended functions correctly but too late or too early. The issue was more related to data inaccuracies and errors in the merging process. (d) value: The software failure incident did involve a value issue where the system performed its intended functions incorrectly by listing the best-selling books inaccurately due to errors in data merging and processing [87968]. (e) byzantine: The software failure incident did not exhibit a byzantine behavior where the system behaved erroneously with inconsistent responses and interactions. The errors were more related to data processing and merging issues. (f) other: The other behavior exhibited by the software failure incident was related to errors in data ingestion and combination from different sources, leading to substantial and intermittent inaccuracies in the lists of best-selling books [87968].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence unknown The consequence of the software failure incident described in the article is mainly related to the accuracy of the best-selling book lists published by The Washington Post. The software errors caused the lists to be inaccurate, with books being listed in the wrong order and some sales from top-selling titles being ignored. This impacted the reliability and correctness of the information provided to readers interested in the best-selling books [87968]. There were no direct consequences such as death, harm, impact on basic needs, property loss, or delays mentioned in the article. The consequences were more related to data accuracy and information reliability.
Domain information (a) The software failure incident was related to the industry of information, specifically in the production and distribution of information. The incident involved The Washington Post's lists of best-selling books, where the software that merged data from NPD BookScan and Amazon began experiencing errors, leading to inaccurate lists being published [Article 87968].

Sources

Back to List