Incident: AlphaGo Software Vulnerability Exploited by Lee Sedol, Impacting Google

Published Date: 2016-03-14

Postmortem Analysis
Timeline 1. The software failure incident happened on March 13, 2016 [41925].
System unknown
Responsible Organization 1. Lee Sedol identified weaknesses in the AlphaGo software that allowed him to exploit them during the match [41925]. 2. AlphaGo, the artificial intelligence program developed by Google's DeepMind, was responsible for the software failure incident as it exhibited a bug or lack of ability to deal with surprises when faced with unexpected moves from Lee Sedol [41925].
Impacted Organization 1. Lee Sedol, the human Go player, was impacted by the software failure incident as he identified weaknesses in the AlphaGo program and exploited them to secure a victory [41925].
Software Causes 1. The software failure incident was caused by weaknesses in the AlphaGo program that were identified and exploited by human Go player Lee Sedol, including the program's inability to deal with surprises and difficulties when playing with a black stone [41925].
Non-software Causes 1. Lee Sedol identified weaknesses in the software, indicating potential flaws in the AI program's decision-making capabilities [41925].
Impacts 1. The software failure incident, where Lee Sedol identified weaknesses in Google's AlphaGo program during a Go match, led to him scoring his first victory over the AI, breaking the winning streak of the machine [41925]. 2. Lee Sedol's win highlighted that humans could still outmaneuver machines by exploiting bugs or weaknesses in the software, showcasing the limitations of the AI program [41925]. 3. The incident demonstrated that the AI program, AlphaGo, lacked the ability to deal with surprises and unexpected moves, indicating a flaw in its decision-making process [41925]. 4. The software failure incident impacted the reputation of AlphaGo and DeepMind, as it showed that the program was not infallible and could be defeated by a skilled human player who identified bugs in the AI software [41925].
Preventions 1. Conducting thorough testing and quality assurance processes to identify and address bugs or weaknesses in the software before deployment [41925]. 2. Implementing continuous monitoring and feedback mechanisms to detect and address any unexpected behavior or bugs in real-time [41925]. 3. Enhancing the software's ability to handle surprises and unexpected moves to prevent critical mistakes during operation [41925].
Fixes 1. Conducting a thorough analysis of the weaknesses identified in the software by Lee Sedol to address potential bugs or lack of ability to deal with surprises [41925] 2. Implementing improvements in the program's ability to handle unexpected moves and surprises to prevent similar incidents in the future [41925]
References 1. Lee Sedol [41925] 2. Demis Hassabis [41925] 3. Marc Zuckerberg [41925] 4. DeepMind [41925] 5. Google [41925] 6. Facebook [41925] 7. British Go Association [41925]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident having happened again at one_organization: The incident of software failure happened again within the same organization, Google, with its AlphaGo program. In the article, it is mentioned that during the match between the AlphaGo program and world champion Lee Sedol, Lee was able to find weaknesses in the software and exploit them to secure a victory. Lee identified two bugs in the AI software that he was able to exploit during the game [41925]. (b) The software failure incident having happened again at multiple_organization: There is no information in the provided articles about the software failure incident happening again at other organizations or with their products and services.
Phase (Design/Operation) operation (a) The articles do not provide any information about a software failure incident related to the design phase, where contributing factors introduced by system development, system updates, or procedures to operate or maintain the system led to the failure. Hence, there is no specific mention of a failure in the design phase in the provided articles. (b) The software failure incident related to the operation phase is highlighted in the articles. Lee Sedol, the human Go player, was able to score his first victory over the AlphaGo computer program by exploiting weaknesses he identified in the software during operation. He mentioned that when he made unexpected moves, AlphaGo responded as if the program had a bug, indicating that the machine lacked the ability to deal with surprises. This incident demonstrates a failure in the operation phase where the system was unable to handle unexpected inputs effectively, leading to a vulnerability that was exploited by the human player [41925].
Boundary (Internal/External) within_system (a) within_system: The software failure incident mentioned in the articles is primarily attributed to factors originating from within the system. Lee Sedol, the human Go player, was able to find weaknesses in the AlphaGo software developed by Google's DeepMind. He identified bugs in the AI software that he could exploit during the match, leading to his victory over the artificial intelligence program [41925]. Additionally, AlphaGo made a mistake around move 79 in one of the games but only realized it by move 87, indicating an internal issue within the software that affected its performance [41925]. These instances highlight how the software's internal weaknesses contributed to the failure experienced during the matches.
Nature (Human/Non-human) human_actions (a) The software failure incident occurring due to non-human actions: - The software failure incident in this case was not due to non-human actions but rather due to Lee Sedol, the human Go player, finding weaknesses in the AlphaGo software [41925]. (b) The software failure incident occurring due to human actions: - Lee Sedol, the human Go player, was able to exploit weaknesses in the AlphaGo software, indicating that the failure was due to contributing factors introduced by human actions [41925].
Dimension (Hardware/Software) software (a) The articles do not mention any software failure incident occurring due to contributing factors originating in hardware. (b) The software failure incident mentioned in the articles is related to weaknesses found in the software used for the AlphaGo program during the Go match between Lee Sedol and the AlphaGo computer. Lee Sedol identified weaknesses in the program, such as the machine's inability to deal with surprises and difficulties when playing with a black stone, which he exploited to secure a victory [41925].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident in the articles is non-malicious. Lee Sedol, the human Go player, identified weaknesses in the AlphaGo software developed by Google's DeepMind during a Go match. He exploited these weaknesses to secure a victory over the artificial intelligence program. Sedol mentioned that when he made unexpected moves, AlphaGo responded as if it had a bug, indicating a lack of ability to deal with surprises. Additionally, Sedol noted that AlphaGo had more difficulty when playing with a black stone, suggesting vulnerabilities in the program [41925].
Intent (Poor/Accidental Decisions) accidental_decisions The intent of the software failure incident in the articles is related to accidental_decisions. Lee Sedol, the human Go player, was able to find weaknesses in the AlphaGo software during the match. He identified two bugs in the AI software that he was able to exploit, indicating that the machine lacked the ability to deal with surprises and had more difficulty when playing with a black stone [41925]. This incident highlights how human intuition and unexpected moves led to the software's failure during the game.
Capability (Incompetence/Accidental) unknown (a) The software failure incident occurring due to development incompetence: - The incident mentioned in the articles does not indicate a software failure due to development incompetence. Instead, it highlights the human player, Lee Sedol, finding weaknesses in the software and exploiting them to achieve victory over the AlphaGo program [41925]. (b) The software failure incident occurring accidentally: - The incident does not directly point to a software failure occurring accidentally. It mainly focuses on the strategic gameplay between the human player and the AlphaGo program, with Lee Sedol identifying weaknesses in the software and leveraging them to secure a win [41925].
Duration temporary The software failure incident discussed in the articles is temporary. Lee Sedol, the human Go player, was able to find weaknesses in the AlphaGo software during the match. He identified two bugs in the AI software that he was able to exploit, leading to his victory over the artificial intelligence program [41925]. Additionally, during the match, AlphaGo made a mistake around move 79 but only realized it by move 87, indicating a temporary failure in its decision-making process [41925].
Behaviour omission, value, other (a) crash: Failure due to system losing state and not performing any of its intended functions - The software failure incident related to a crash is not explicitly mentioned in the provided article. (b) omission: Failure due to system omitting to perform its intended functions at an instance(s) - Lee Sedol identified weaknesses in the software, indicating that the system omitted to perform its intended functions correctly [41925]. (c) timing: Failure due to system performing its intended functions correctly, but too late or too early - The software failure incident related to timing is not explicitly mentioned in the provided article. (d) value: Failure due to system performing its intended functions incorrectly - Lee Sedol exploited weaknesses in the software, indicating that the system performed its intended functions incorrectly [41925]. (e) byzantine: Failure due to system behaving erroneously with inconsistent responses and interactions - The software failure incident related to a byzantine behavior is not explicitly mentioned in the provided article. (f) other: Failure due to system behaving in a way not described in the (a to e) options; What is the other behavior? - The other behavior observed in the software failure incident is the system's inability to deal with surprises, as indicated by Lee Sedol making an unexpected move and the system responding as if it had a bug [41925].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence no_consequence (a) death: People lost their lives due to the software failure - There is no mention of any deaths resulting from the software failure incident reported in the articles. [41925]
Domain information, knowledge, entertainment, other (a) The failed system in the incident was related to the information industry, specifically in the context of artificial intelligence and gaming. The software failure incident involved Google's AlphaGo program, developed by DeepMind, which is an artificial intelligence program designed to play the ancient Chinese board game Go. The incident highlighted the competition between human intelligence and machine learning capabilities in the realm of gaming and artificial intelligence [41925]. (i) The incident also relates to the knowledge industry as it involves advancements in artificial intelligence research and technology. The development of AlphaGo by Google's DeepMind represents a significant milestone in AI research, particularly in the field of game-playing algorithms. The incident showcased the capabilities of AI systems to learn and adapt strategies through neural networks and deep learning techniques [41925]. (m) The software failure incident can be categorized under the "other" industry as it pertains to the intersection of technology, artificial intelligence, and gaming. The incident does not fit neatly into the traditional industry categories but rather represents a unique case of software failure within the context of cutting-edge AI development and gaming competitions [41925].

Sources

Back to List