Incident: Facebook's Child Nudity Detection Software Failure.

Published Date: 2018-10-24

Postmortem Analysis
Timeline 1. The software failure incident of Facebook's child nudity detection and grooming AI happened over the last year [76749]. Therefore, the incident occurred between October 2017 and October 2018.
System The software failure incident mentioned in the article did not involve a system failure.
Responsible Organization 1. Facebook [76749]
Impacted Organization 1. Children - The software failure incident impacted children as it involved the removal of 8.7 million user images of child nudity and the detection of child exploitation on Facebook [76749].
Software Causes 1. The failure incident was caused by the imperfections in Facebook's machine learning programs used to detect child nudity and exploitation content, leading to mistakes in content moderation [76749].
Non-software Causes 1. Lack of proper moderation and oversight: The incident highlights the need for human moderators and trained teams to review content, as well as the importance of prioritizing and efficiently handling problematic content [76749]. 2. Pressure from regulators and lawmakers: Facebook's response to the incident was influenced by external pressure to speed up the removal of extremist and illicit material, indicating the impact of regulatory and legislative pressures on the company's actions [76749]. 3. Challenges with encrypted chat apps and 'dark web' sites: The failure incident also points out the limitations in addressing issues related to child pornography originating from encrypted chat apps and secretive 'dark web' sites, highlighting the difficulty in monitoring and addressing content in such platforms [76749].
Impacts unknown
Preventions 1. Implementing more rigorous testing procedures during the development phase to identify and address any potential issues before deployment [76749]. 2. Conducting regular audits and reviews of the machine learning algorithms to ensure they are functioning as intended and not producing false positives or false negatives [76749]. 3. Enhancing the transparency of the software by providing clear explanations of how the AI systems work and what criteria they use to flag content, which can help in building trust with users and reducing potential errors [76749]. 4. Collaborating with external organizations and experts in child safety and exploitation to receive feedback and guidance on improving the effectiveness of the software in detecting and removing harmful content [76749].
Fixes 1. Implementing more robust and accurate machine learning algorithms to better identify and flag child nudity and exploitative content on Facebook [76749]. 2. Enhancing the child grooming detection system by considering additional factors and improving the accuracy of identifying potential threats [76749]. 3. Developing software to prioritize reports of child pornography and exploitation for law enforcement agencies, in collaboration with organizations like the National Center for Missing and Exploited Children (NCMEC) [76749]. 4. Educating tech companies on addressing the issue of encrypted chat apps and 'dark web' sites where new child pornography originates, and encouraging them to find creative solutions to tackle this crucial blind spot [76749].
References 1. Facebook's global head of safety Antigone Davis 2. Michelle DeLaune, chief operating officer at the National Center for Missing and Exploited Children (NCMEC) [76749]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident related to child nudity detection and removal has happened again within the same organization, Facebook. The article mentions that Facebook had not previously disclosed data on child nudity removals, indicating that this incident is not the first of its kind within the company [76749]. Additionally, the article highlights that Facebook is exploring applying the same technology to its Instagram app, suggesting a continuation of the incident within the organization.
Phase (Design/Operation) design, operation (a) The software failure incident related to the design phase can be seen in the development and implementation of Facebook's machine learning tool to automatically flag images containing child nudity. This tool was rolled out over the last year and is designed to identify images that contain both nudity and a child, enabling increased enforcement of Facebook's policies on photos showing minors in a sexualized context [76749]. (b) The software failure incident related to the operation phase is evident in the challenges faced by Facebook's machine learning programs, which sift through billions of pieces of content posted by users each day. There have been complaints from news agencies and advertisers about Facebook's automated systems wrongly blocking their posts, indicating issues with the operation or misuse of the system [76749].
Boundary (Internal/External) within_system (a) within_system: The software failure incident related to the removal of child nudity images on Facebook was primarily due to factors originating from within the system. Facebook implemented machine learning tools and AI systems to automatically flag and remove images containing child nudity and sexual exploitation content. The AI system was trained on a collection of nude adult photos and clothed children photos to proactively detect such content when uploaded. Additionally, Facebook's global head of safety mentioned that the machine learning technology helps prioritize and efficiently queue problematic content for review by the company's trained team of moderators [76749]. (b) outside_system: The articles do not provide specific information about the software failure incident being caused by factors originating from outside the system.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident related to non-human actions in this case is the removal of 8.7 million user images of child nudity by Facebook's machine learning tool that automatically flags such photos [76749]. This incident was a result of the machine learning tool identifying images containing both nudity and a child, leading to increased enforcement of Facebook's policies on photos showing minors in a sexualized context. The AI system rolled out by Facebook over the last year played a significant role in detecting and removing such content without direct human involvement. (b) On the other hand, the software failure incident related to human actions involves the reinstatement of the iconic photograph of a naked girl fleeing a napalm attack during the Vietnam War on Facebook. This incident occurred in 2016 when Facebook deleted the photograph from the pages of several Norwegian authors and media outlets, including a top-selling newspaper, following outrage from Norway's prime minister and various groups. The deletion of the photograph was a direct result of human actions, specifically the decision-making process within Facebook that led to the removal of the historically significant image [76749].
Dimension (Hardware/Software) software (a) The articles do not mention any software failure incident related to hardware issues [76749]. (b) The software failure incident mentioned in the articles is related to the removal of child nudity images on Facebook. The failure was due to contributing factors that originate in software, specifically the machine learning tool and AI system developed by Facebook to automatically flag and remove images containing child nudity and sexual exploitation content [76749].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident related to the removal of child nudity images on Facebook does not appear to be malicious. The incident involves the use of machine learning tools and AI to automatically flag and remove images containing child nudity or sexual exploitation. Facebook implemented these tools to enforce its policies against such content and to prioritize the removal of problematic content efficiently [76749]. The software was designed to proactively detect and remove such content, indicating a non-malicious intent to protect users and enforce community standards.
Intent (Poor/Accidental Decisions) poor_decisions, accidental_decisions (a) The intent of the software failure incident related to poor_decisions: - Facebook's previous reliance on users or its adult nudity filters to catch child images before the implementation of the machine learning tool to automatically flag such photos [76749]. - Facebook's rules for years banning even family photos of lightly clothed children uploaded with 'good intentions' due to concerns about potential abuse of such images [76749]. (b) The intent of the software failure incident related to accidental_decisions: - Facebook's machine learning tool mistakenly blocking posts from news agencies and advertisers due to imperfections in the system [76749]. - Acknowledgment by Facebook's global head of safety that the child safety systems might make mistakes, but users could appeal [76749].
Capability (Incompetence/Accidental) accidental (a) The software failure incident related to development incompetence is not evident from the provided articles. (b) The software failure incident related to accidental factors is highlighted in the article where Facebook mistakenly deleted the iconic photograph of a naked girl fleeing a napalm attack during the Vietnam War. The deletion occurred after Norway's prime minister and many Norwegian authors and media groups expressed outrage over the removal of the image. This accidental deletion of historically significant content showcases a failure introduced accidentally [76749].
Duration temporary The software failure incident described in the articles is more related to a temporary failure rather than a permanent one. The incident involves the implementation of a machine learning tool by Facebook to automatically flag images containing child nudity and to detect grooming behavior towards minors. This tool was rolled out over the last year and has led to increased enforcement of Facebook's policies regarding child exploitation content. The temporary failure in this case would be due to contributing factors introduced by certain circumstances, such as the need for improved detection and removal of harmful content on the platform, rather than being a permanent failure inherent to the software itself. [Cited Article: 76749]
Behaviour omission (a) crash: The articles do not mention any software crash incidents. (b) omission: The software system mentioned in the articles is designed to automatically flag and remove images containing child nudity on Facebook. It is mentioned that the system helps in prioritizing and efficiently queuing problematic content for the company's team of reviewers. The system is proactive in detecting child nudity and exploitation content when uploaded, with 99% of the violating content being removed before anyone reports it [76749]. (c) timing: The articles do not mention any timing-related failures of the software system. (d) value: The software system mentioned in the articles is focused on identifying and removing content that violates Facebook's policies related to child nudity and exploitation. It is mentioned that the system has led to more removals of such content and that exceptions are made for art and historical images [76749]. (e) byzantine: The articles do not mention any byzantine behavior of the software system. (f) other: The software system mentioned in the articles is designed to catch users engaged in 'grooming,' which involves befriending minors for sexual exploitation. The system evaluates factors such as how many people have blocked a particular user and whether that user quickly attempts to contact many children. Additionally, the system is used to prioritize reports of child exploitation content for law enforcement agencies. The software is also being explored for application on Instagram [76749].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence no_consequence, theoretical_consequence, unknown (a) unknown (b) unknown (c) unknown (d) unknown (e) unknown (f) unknown (g) no_consequence (h) theoretical_consequence (i) unknown
Domain information The software failure incident reported in the articles is related to the industry of information (a). The incident involves Facebook's use of machine learning tools to automatically flag and remove images containing child nudity and exploitation on its platform. The AI system rolled out by Facebook is aimed at identifying and removing such content to enforce the company's policies against child sexual exploitation [Article 76749].

Sources

Back to List