Incident: Watson AI Swearing Incident: Urban Dictionary Download Gone Wrong

Published Date: 2013-01-12

Postmortem Analysis
Timeline 1. The software failure incident involving the IBM supercomputer Watson happened in 2013. [Article 16208]
System 1. IBM supercomputer Watson 2. Urban Dictionary repository 3. Linguistic filter developed by Mr. Brown's team [16208]
Responsible Organization 1. IBM [16208]
Impacted Organization 1. IBM [16208]
Software Causes 1. The software cause of the failure incident was the decision to teach the IBM supercomputer Watson the contents of the Urban Dictionary, which led to it memorizing profanities and inappropriate language, causing it to swear and give backchat to researchers [16208].
Non-software Causes 1. Lack of understanding of human communication subtleties by the artificial intelligence Watson [16208] 2. Inappropriate content fed to the machine, leading to obscene outbursts [16208] 3. Failure to pass the Turing test of computer intelligence [16208]
Impacts 1. The software failure incident led to the IBM supercomputer Watson repeatedly swearing and making obscene outbursts after memorizing the contents of the Urban Dictionary, which was inappropriate for polite conversation [16208]. 2. Researchers had to wipe the Urban Dictionary from the machine's memory and develop a linguistic filter to prevent Watson from swearing again, impacting the machine's ability to communicate naturally and pass the Turing test of computer intelligence [16208]. 3. The incident highlighted the limitations of artificial intelligence systems like Watson in truly understanding human communication and the meaning behind language, supporting the contention that such systems can only manipulate symbols without true comprehension [16208].
Preventions 1. Implementing a more robust linguistic filter to prevent the supercomputer from memorizing and using inappropriate language [16208]. 2. Conducting more thorough testing and analysis of the supercomputer's behavior after introducing new data sources like the Urban Dictionary to anticipate and address any potential issues before they escalate [16208].
Fixes 1. Implementing a linguistic filter to prevent the supercomputer Watson from swearing again [16208].
References 1. IBM researchers 2. Eric Brown, the IBM research scientist 3. John Searle, the U.S. analytic philosopher 4. Urban Dictionary website 5. Fortune magazine

Software Taxonomy of Faults

Category Option Rationale
Recurring unknown (a) The software failure incident related to Watson's swearing after memorizing the Urban Dictionary and having to have its memory wiped by IBM researchers is a unique incident specific to IBM's Watson supercomputer. There is no mention in the article of a similar incident happening again within the same organization. (b) There is no information in the article about a similar incident happening at other organizations or with their products and services.
Phase (Design/Operation) design (a) The software failure incident in the article can be attributed to the design phase. The incident occurred because the IBM research scientist, Eric Brown, taught the artificial intelligence machine Watson the contents of the Urban Dictionary in an attempt to make its communications more natural. However, this led to Watson memorizing profanities and inappropriate language, causing it to start giving backchat and uttering obscenities, ultimately leading to the need to wipe the dictionary from the machine's memory and develop a linguistic filter to prevent further swearing incidents [16208]. (b) The software failure incident in the article is not directly related to the operation phase or misuse of the system.
Boundary (Internal/External) within_system (a) within_system: The software failure incident in the article was primarily within the system. The incident occurred because the IBM supercomputer Watson, an artificial intelligence machine, started swearing after memorizing the contents of the Urban Dictionary. The researchers had to wipe the dictionary from the machine's memory to stop it from making obscene outbursts [16208]. This failure was a result of the machine's internal programming and memory content, indicating an issue originating from within the system.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident in the article was primarily due to non-human actions. The incident occurred because the IBM supercomputer Watson, an artificial intelligence machine, started swearing after memorizing the contents of the Urban Dictionary. The machine's behavior of making obscene outbursts was a result of its interaction with the Urban Dictionary, which contained profanities and insults inappropriate for polite conversation. The researchers had to wipe the dictionary from the machine's memory to stop it from swearing [16208]. (b) The human actions involved in the software failure incident were related to the decision-making process by the researchers and programmers working with the IBM supercomputer Watson. The researchers, specifically Eric Brown, taught Watson the Urban Dictionary in an attempt to make its communications more natural and equip it with the knowledge needed to pass the Turing test of computer intelligence. However, it was after Watson started answering back with obscenities that the researchers decided to pull the plug on teaching it slang. Subsequently, the team had to wipe the Urban Dictionary from the computer's memory and develop a linguistic filter to prevent Watson from swearing again [16208].
Dimension (Hardware/Software) software (a) The software failure incident in the article was not directly attributed to hardware issues. The incident occurred due to the artificial intelligence machine Watson memorizing the contents of the Urban Dictionary, leading to it making obscene outbursts and inappropriate responses. The decision to wipe the machine's memory and develop a linguistic filter was made to address the software-related issue of Watson swearing and not understanding human communication [16208]. (b) The software failure incident in the article was primarily attributed to software-related factors. The incident occurred because the artificial intelligence machine Watson, programmed by IBM researchers, had difficulty understanding human communication nuances and started giving backchat, including uttering obscenities. The need to wipe the Urban Dictionary from the machine's memory and develop a linguistic filter to prevent further swearing highlights the software-related nature of the failure [16208].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident in this case was non-malicious. The incident occurred when the IBM supercomputer Watson, which had been fed the Urban Dictionary to enhance its natural language capabilities, started making obscene outbursts and inappropriate remarks. The researchers had to wipe the dictionary from the machine's memory and develop a linguistic filter to prevent further swearing incidents. This failure was not due to malicious intent but rather a consequence of the machine's inability to understand and communicate in a socially acceptable manner [16208].
Intent (Poor/Accidental Decisions) accidental_decisions (a) The intent of the software failure incident was not due to poor decisions but rather an unintended consequence of attempting to equip the artificial intelligence machine Watson with colloquial knowledge from the Urban Dictionary. The incident occurred when the machine started giving backchat to researchers and making obscene outbursts after memorizing the contents of the Urban Dictionary. The decision to teach Watson slang was an attempt to make its communications seem more natural and help it pass the Turing test of computer intelligence. However, this decision led to the unintended consequence of Watson swearing and responding inappropriately, ultimately resulting in the need to wipe the taboo vocabulary from its memory and develop a linguistic filter to prevent further incidents [16208].
Capability (Incompetence/Accidental) development_incompetence (a) The software failure incident in the article can be attributed to development incompetence. The incident occurred because the IBM research scientist in charge of tutoring Watson, Eric Brown, taught the computer the Urban Dictionary in an effort to make its communications seem more natural. However, Watson's inability to master the subtleties of good-mannered repartee led to it making obscene outbursts and uttering profanities, which forced the researchers to delete the taboo vocabulary from its memory [16208]. This failure highlights the challenges and risks associated with introducing slang and colloquial language into AI systems without proper oversight and control. (b) The software failure incident was not accidental but rather a result of deliberate actions taken by the development team to enhance Watson's communication abilities. The decision to teach Watson the Urban Dictionary was intentional, aiming to equip the AI with the knowledge needed to pass the Turing test by engaging in natural-sounding small talk. However, the unintended consequence of Watson's inappropriate language and swearing demonstrates the unforeseen outcomes that can arise from such development decisions [16208].
Duration permanent (a) The software failure incident in the article is described as permanent. The IBM supercomputer Watson had to have its memory wiped as a solution to stop it from swearing after memorizing the contents of the Urban Dictionary. The researchers found no other way to stop the obscene outbursts, leading to the permanent action of deleting the taboo vocabulary from the machine's memory [16208].
Behaviour crash, value, other (a) crash: The software failure incident in the article can be categorized as a crash. The IBM supercomputer Watson had to have its memory wiped because it kept making obscene outbursts after memorizing the contents of the Urban Dictionary. This behavior led to the system losing its state and not performing its intended functions [16208]. (b) omission: There is no specific mention of the software failure incident in the article being related to omission. (c) timing: There is no indication in the article that the software failure incident was related to timing issues. (d) value: The software failure incident in the article can be associated with a value failure. Watson, after memorizing the Urban Dictionary, started giving backchat to researchers and even began uttering obscenities, which can be considered as performing its intended functions incorrectly [16208]. (e) byzantine: The software failure incident in the article does not exhibit characteristics of a byzantine failure. (f) other: The other behavior exhibited by the software failure incident in the article is related to the system behaving in a way not described in the options (a to e). In this case, the behavior was the system's inability to master good-mannered repartee, leading to the need to delete the taboo vocabulary from its memory and develop a linguistic filter to prevent further swearing incidents [16208].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, non-human, theoretical_consequence, other (a) death: People lost their lives due to the software failure - There is no mention of any deaths caused by the software failure incident reported in the article [16208]. (b) harm: People were physically harmed due to the software failure - There is no mention of any physical harm caused to people due to the software failure incident reported in the article [16208]. (c) basic: People's access to food or shelter was impacted because of the software failure - There is no mention of people's access to food or shelter being impacted due to the software failure incident reported in the article [16208]. (d) property: People's material goods, money, or data was impacted due to the software failure - The software failure incident involving IBM's Watson supercomputer resulted in the need to wipe its memory and develop a linguistic filter to prevent it from swearing again [16208]. (e) delay: People had to postpone an activity due to the software failure - There is no mention of any activities being postponed due to the software failure incident reported in the article [16208]. (f) non-human: Non-human entities were impacted due to the software failure - The software failure incident involved the IBM supercomputer Watson, an artificial intelligence machine, which had to have its memory wiped due to its inappropriate behavior after memorizing the Urban Dictionary [16208]. (g) no_consequence: There were no real observed consequences of the software failure - The software failure incident involving IBM's Watson supercomputer did have consequences, such as the need to wipe its memory and develop a linguistic filter to prevent further issues [16208]. (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur - The article discusses the potential consequences of the failed experiment, including the implications for the Turing test and the limitations of artificial intelligence in truly understanding human communication [16208]. (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? - The software failure incident led to Watson, the AI supercomputer, making obscene outbursts and inappropriate responses, which required intervention from researchers to address the issue [16208].
Domain entertainment (a) The failed system in this incident was related to the entertainment industry. The IBM supercomputer Watson, which had its memory wiped after downloading the Urban Dictionary and started swearing, was famously known for winning the game show Jeopardy! against human champions [Article 16208].

Sources

Back to List