Incident: Facebook Bug Incorrectly Marks Coronavirus Posts as Spam

Published Date: 2020-03-17

Postmortem Analysis
Timeline 1. The software failure incident of Facebook incorrectly marking posts as spam due to a bug happened around March 16, 2020, as per the article published on March 17, 2020 [97470].
System 1. Anti-spam system - Facebook's anti-spam system failed, causing posts on various topics, including coronavirus-related content, to be incorrectly marked as spam [97470].
Responsible Organization 1. Facebook's software issue [97470]
Impacted Organization 1. Facebook and Instagram users [97470]
Software Causes 1. A software bug in an anti-spam system was responsible for incorrectly marking posts on topics, including coronavirus, as spam on Facebook [97470].
Non-software Causes 1. Contract workers who review content were sent home due to the outbreak [97470]. 2. Facebook announced sending home all contract workers who perform content review services for public health reasons [97470]. 3. Facebook experienced limitations in their approach due to the situation, leading to longer response times and more mistakes [97470]. 4. Facebook gave $1,000 bonuses to its employees to support them during the coronavirus pandemic [97470]. 5. Facebook offered $100m to help small businesses cope with the economic impact of the outbreak [97470]. 6. Facebook, along with other tech giants, asked staff to work from home due to the virus spread [97470].
Impacts 1. Posts on various topics, including coronavirus, were incorrectly marked as spam on Facebook and Instagram, causing widespread complaints from users [97470]. 2. Notifications were sent to users stating that articles from prominent news organizations like the BBC, Axios, and The Atlantic had violated the company's community guidelines [97470]. 3. Contract workers who review content were sent home due to the outbreak, potentially leading to longer response times and more mistakes in content moderation [97470].
Preventions 1. Implementing thorough testing procedures before deploying any changes to the anti-spam system could have potentially prevented the software failure incident [97470]. 2. Conducting regular audits and reviews of the anti-spam system to identify and address any potential bugs or issues proactively could have helped prevent such incidents [97470]. 3. Providing continuous training and support to the contract workers who review content to ensure they are aware of the system's guidelines and limitations could have contributed to preventing the incorrect marking of posts as spam [97470].
Fixes 1. Implementing a fix for the bug in the anti-spam system responsible for incorrectly marking posts as spam on Facebook [97470].
References 1. Facebook's head of safety 2. Guy Rosen, Facebook's vice president for integrity 3. Mark Zuckerberg, Facebook's chief executive 4. Facebook's blog post 5. Internal note from Mark Zuckerberg 6. Facebook's chief operating officer Sheryl Sandberg 7. Twitter post by Guy Rosen 8. Complaints from Facebook and Instagram users 9. Notifications received by users 10. Announcement by Facebook regarding contract workers 11. Announcement by Facebook regarding bonuses and support for employees 12. Announcement by Facebook regarding support for small businesses 13. Facebook post by Sheryl Sandberg 14. Tech giants including Google and Twitter

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident related to incorrectly marking posts as spam on Facebook due to a bug in the anti-spam system has happened again within the same organization. Facebook acknowledged that this incident was caused by a software issue, specifically a bug in their anti-spam system [97470]. This incident led to posts on various topics, including coronavirus-related content, being wrongly marked as spam, resulting in widespread complaints from users. The issue arose after Facebook had announced sending home contract workers who perform content review services due to the outbreak, indicating a recurrence of the software failure incident within the organization.
Phase (Design/Operation) design (a) The software failure incident mentioned in the articles can be attributed to the design phase. Facebook acknowledged that the incorrectly marked spam posts, including those related to coronavirus, were due to a bug in their anti-spam system. This bug introduced by the system development led to posts being wrongly flagged as spam, causing widespread complaints from users [97470].
Boundary (Internal/External) within_system (a) within_system: The software failure incident reported in the article is attributed to a bug in Facebook's anti-spam system, which incorrectly marked posts on various topics, including coronavirus, as spam. Facebook's head of safety mentioned, "This is a bug in an anti-spam system" [97470]. This indicates that the failure originated from within the system itself, specifically from the anti-spam system. (b) outside_system: The article does not provide information suggesting that the software failure incident was caused by contributing factors originating from outside the system.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident in this case was attributed to a bug in an anti-spam system on Facebook, which incorrectly marked posts on various topics, including coronavirus, as spam. This bug was a non-human action that led to the failure [97470]. (b) Additionally, human actions were involved in the response to the incident. Facebook announced sending home contract workers who review content due to the outbreak, which could have contributed to longer response times and potential mistakes in handling such issues [97470].
Dimension (Hardware/Software) software (a) The software failure incident reported in the article is attributed to a bug in an anti-spam system on Facebook, which incorrectly marked posts on various topics, including coronavirus, as spam. This bug led to widespread complaints from users and required the restoration of all incorrectly removed posts [97470]. (b) The software failure incident is specifically mentioned to be caused by a bug in an anti-spam system on Facebook, indicating that the contributing factor for this failure originates in the software itself [97470].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident mentioned in the article is non-malicious. Facebook attributed the issue to a bug in their anti-spam system, which incorrectly marked posts on various topics, including coronavirus, as spam. The company's head of safety stated that it was a software issue and not a deliberate act of malicious intent [97470].
Intent (Poor/Accidental Decisions) accidental_decisions (a) The software failure incident mentioned in the article was not due to poor decisions but rather a bug in an anti-spam system on Facebook that incorrectly marked posts, including those related to coronavirus, as spam. This bug led to widespread complaints from users and required the restoration of the incorrectly removed posts [97470].
Capability (Incompetence/Accidental) accidental (a) The software failure incident related to development incompetence is not explicitly mentioned in the provided article. (b) The software failure incident related to accidental factors is evident in the article. Facebook attributed the issue of posts on topics like coronavirus being wrongly marked as spam to a software bug in their anti-spam system. This indicates that the failure was accidental and not intentional [97470].
Duration temporary (a) The software failure incident mentioned in the article was temporary. It was caused by a bug in an anti-spam system on Facebook that incorrectly marked posts, including those related to coronavirus, as spam. Facebook acknowledged the issue and restored all the incorrectly removed posts, indicating that it was a specific incident caused by the bug in the system [97470].
Behaviour value, other (a) crash: The software failure incident reported in the article is not related to a crash where the system loses state and does not perform any of its intended functions. [97470] (b) omission: The software failure incident is not due to the system omitting to perform its intended functions at an instance(s). It is more related to incorrectly marking posts as spam. [97470] (c) timing: The software failure incident is not due to the system performing its intended functions correctly but too late or too early. It is more about incorrectly marking posts as spam. [97470] (d) value: The software failure incident is related to the system performing its intended functions incorrectly by marking posts on topics including coronavirus as spam. This behavior was attributed to a bug in an anti-spam system. [97470] (e) byzantine: The software failure incident is not related to the system behaving erroneously with inconsistent responses and interactions. It is more about incorrectly marking posts as spam. [97470] (f) other: The software failure incident behavior is specifically related to a bug in an anti-spam system that led to posts on various topics, including coronavirus, being wrongly marked as spam. This behavior resulted in widespread complaints from users. [97470]

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence delay (a) death: People lost their lives due to the software failure (b) harm: People were physically harmed due to the software failure (c) basic: People's access to food or shelter was impacted because of the software failure (d) property: People's material goods, money, or data was impacted due to the software failure (e) delay: People had to postpone an activity due to the software failure (f) non-human: Non-human entities were impacted due to the software failure (g) no_consequence: There were no real observed consequences of the software failure (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? The articles do not mention any consequences related to death, harm, basic needs, property loss, or non-human entities due to the software failure incident. The main consequence discussed in the articles is related to potential delays and mistakes in content review due to the software bug in Facebook's anti-spam system [97470].
Domain information (a) The failed system in this incident was related to the industry of information, specifically social media platforms like Facebook and Instagram where posts on various topics, including coronavirus, were incorrectly marked as spam due to a software bug [97470]. (b) N/A (c) N/A (d) N/A (e) N/A (f) N/A (g) N/A (h) N/A (i) N/A (j) N/A (k) N/A (l) N/A (m) N/A

Sources

Back to List