Incident: Failure Title: Alibaba Cloud Suspended for Log4Shell Security Flaw Incident

Published Date: 2021-12-21

Postmortem Analysis
Timeline 1. The software failure incident involving Alibaba Cloud and the Log4shell security flaw happened in December 2021 as reported in Article [122008, 122105].
System 1. Alibaba Cloud's reporting system 2. Log4j software (specifically Log4j2 version) [Article 122008, Article 122105]
Responsible Organization 1. Alibaba Cloud engineer who discovered the Log4shell security flaw [Article 122008] 2. Alibaba Cloud for not promptly reporting vulnerabilities in Apache Log4j2 to China's telecommunications regulator [Article 122105]
Impacted Organization 1. Alibaba Cloud Computing, a subsidiary of Alibaba Group [122008, 122105] 2. Ministry of Industry and Information Technology (MIIT) in China [122008, 122105]
Software Causes 1. The software cause of the failure incident was the Log4Shell security flaw in the widely used Log4j software, which was discovered by Alibaba Cloud engineers [Article 122008, Article 122105].
Non-software Causes 1. Lack of prompt reporting of cybersecurity vulnerability to the Ministry of Industry and Information Technology (MIIT) by Alibaba Cloud [122008, 122105] 2. Failure to effectively support the ministry's efforts to manage cybersecurity threats and vulnerabilities [122008] 3. Beijing's desire to strengthen control over key online infrastructure and data in the name of national security [122008, 122105] 4. Concern at a vulnerability that has triggered a wave of panic among corporations and governments worldwide [122105]
Impacts 1. The Chinese regulator suspended a partnership with Alibaba Cloud due to the company's failure to promptly report the Log4shell security flaw, impacting their information-sharing platform and cybersecurity cooperation [122008, 122105]. 2. The suspension highlighted Beijing's desire to strengthen control over key online infrastructure and data in the name of national security, indicating a shift towards state-backed cloud systems for data storage [122008, 122105]. 3. The Log4shell security flaw in the widely used Log4j software posed a significant risk, potentially leading to remote control of equipment, theft of sensitive information, and interruption of equipment services, causing panic among corporations and governments worldwide [122105].
Preventions 1. Timely Reporting: The software failure incident could have been prevented if Alibaba Cloud promptly reported the Log4shell security flaw to the Ministry of Industry and Information Technology (MIIT) as soon as it was discovered [122008, 122105]. 2. Effective Vulnerability Management: Implementing a robust vulnerability management process within the organization could have helped in identifying and addressing the security flaw in a timely manner, thus preventing the incident [122008, 122105]. 3. Compliance with Regulations: Ensuring compliance with regulations that require Chinese companies to report vulnerabilities in their software to the relevant authorities, such as the MIIT, could have prevented the suspension of the partnership and the associated software failure incident [122008].
Fixes 1. Prompt reporting of vulnerabilities to the relevant authorities, such as the Ministry of Industry and Information Technology (MIIT), to ensure timely action and coordination in addressing security flaws [122008, 122105]. 2. Effective support for the ministry's efforts to manage cybersecurity threats and vulnerabilities, demonstrating proactive engagement in enhancing cybersecurity measures [122008]. 3. Implementation of internal reforms by the company, in this case, Alibaba Cloud Computing, to address the root causes of the failure incident and prevent similar occurrences in the future [122105].
References 1. Chinese Ministry of Industry and Information Technology (MIIT) [Article 122008, Article 122105] 2. Alibaba Cloud Computing [Article 122008, Article 122105] 3. State-backed Chinese media [Article 122008, Article 122105] 4. Apache Software Foundation [Article 122008, Article 122105] 5. Reuters news agency [Article 122008] 6. 21st Century Business Herald [Article 122008, Article 122105] 7. South China Morning Post [Article 122008]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident related to Alibaba Cloud's suspension due to the Log4shell security flaw is specific to Alibaba Cloud, which is a subsidiary of the Chinese tech giant Alibaba Group. This incident occurred within the same organization, where Alibaba Cloud failed to promptly report the vulnerability in the Apache Log4j2 software to the Ministry of Industry and Information Technology (MIIT) [122008, 122105]. (b) The articles do not mention any similar incident happening at other organizations or with their products and services.
Phase (Design/Operation) design, operation (a) The software failure incident related to the design phase can be seen in the articles [122008, 122105]. Alibaba Cloud's suspension by the Chinese regulators was due to the failure to promptly report the Log4shell security flaw in the Apache Log4j software to the Ministry of Industry and Information Technology (MIIT). This failure to report the vulnerability in a timely manner was a design-related issue as it involved the handling of security flaws in the system development and reporting procedures. (b) The software failure incident related to the operation phase can be observed in the articles [122105]. The Ministry of Industry and Information Technology (MIIT) suspended the partnership with Alibaba Cloud due to the company not immediately reporting vulnerabilities in the Apache Log4j2 software to the telecommunications regulator. This failure to report and address the cybersecurity vulnerability promptly was an operational issue as it involved the operation and management of the system in response to security threats.
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident involving Alibaba Cloud's suspension of partnership with the Chinese regulator was primarily due to the company's failure to promptly report the Log4shell security flaw in the Apache Log4j software to the Ministry of Industry and Information Technology (MIIT) [122008, 122105]. Alibaba Cloud discovered the security flaw and reported it to the Apache Software Foundation but did not report it to MIIT in a timely manner, leading to the suspension of the partnership. This failure to report the vulnerability internally within the system contributed to the software failure incident. (b) outside_system: The software failure incident also had contributing factors originating from outside the system. The Chinese government's regulatory actions, such as the suspension of the partnership and the requirement for Chinese companies to report vulnerabilities to MIIT, were external factors influencing the incident [122008, 122105]. Additionally, the broader context of Beijing's desire to strengthen control over online infrastructure and data in the name of national security played a role in shaping the response to the software failure incident.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident in this case was primarily due to non-human actions. The incident was caused by a security flaw in the widely used Log4j software, specifically the Log4Shell vulnerability, which was discovered by Alibaba Cloud's engineers [122008, 122105]. This flaw was not introduced by human actions but was a pre-existing vulnerability in the software. (b) However, human actions also played a role in this incident. The Chinese regulators suspended the partnership with Alibaba Cloud because the company did not promptly report the Log4Shell vulnerability to the Ministry of Industry and Information Technology (MIIT) [122008, 122105]. This delay in reporting the vulnerability was a human action that contributed to the software failure incident.
Dimension (Hardware/Software) software (a) The software failure incident reported in the articles is primarily related to software issues rather than hardware. The incident specifically involves a cybersecurity vulnerability in the widely used Log4j software, which is an open-source logging framework overseen by the Apache Software Foundation. The vulnerability, known as Log4Shell, was discovered by Alibaba Cloud and was related to a flaw in the Log4j software. The suspension of the partnership between Alibaba Cloud and the Chinese regulator was due to Alibaba Cloud's alleged failure to promptly report the Log4Shell security flaw to the Ministry of Industry and Information Technology (MIIT) [122008, 122105]. (b) The software failure incident is directly attributed to a software flaw in the Log4j software, which is a software component widely used for logging events in online services. The vulnerability in Log4j, known as Log4Shell, was considered a significant security flaw by experts, with the potential for remote control of equipment and serious consequences such as theft of sensitive information and service interruptions. Alibaba Cloud discovered the vulnerability and reported it to the Apache Software Foundation, but the issue arose when they allegedly did not report it promptly to the Chinese regulator, leading to the suspension of their partnership [122008, 122105].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident in the articles is non-malicious. The failure occurred due to Alibaba Cloud's failure to promptly report a cybersecurity vulnerability in the Apache Log4j2 software to the Chinese telecommunications regulator, MIIT. This led to the suspension of the information-sharing partnership between Alibaba Cloud and MIIT. The suspension was a result of Alibaba Cloud not reporting the vulnerability in time, rather than any malicious intent to harm the system [122008, 122105].
Intent (Poor/Accidental Decisions) poor_decisions, accidental_decisions (a) The software failure incident related to the partnership suspension between Alibaba Cloud and the Chinese regulator was primarily due to poor decisions made by Alibaba Cloud. The suspension occurred because Alibaba Cloud failed to promptly report the Log4j security flaw to the Ministry of Industry and Information Technology (MIIT) in China, as required. This delay in reporting the vulnerability led to the suspension of the partnership, indicating a poor decision on the part of Alibaba Cloud [122008, 122105]. (b) Additionally, the failure can also be attributed to accidental decisions or mistakes made by Alibaba Cloud. The company did discover the Log4j security flaw and reported it to the Apache Software Foundation for a fix. However, they did not report the vulnerability to MIIT in a timely manner, which was an unintended decision that ultimately led to the suspension of the partnership with the Chinese regulator [122008, 122105].
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident related to development incompetence is evident in the articles. Alibaba Cloud's suspension by the Chinese regulators was due to the company's failure to promptly report the Log4j security flaw to the Ministry of Industry and Information Technology (MIIT) [122008, 122105]. This delay in reporting the vulnerability was seen as a lack of effective support for managing cybersecurity threats and vulnerabilities by the ministry [122008]. The incident highlights a failure in professional competence or timely action by the development organization in addressing critical security issues. (b) The software failure incident related to accidental factors is also present in the articles. Alibaba Cloud discovered a remote code execution vulnerability in the Apache Log4j2 component and reported it to the Apache Software Foundation [122105]. However, the MIIT received a report about the issue from a third party instead of directly from Alibaba Cloud, indicating a potential accidental oversight or miscommunication in the reporting process [122105]. This accidental factor may have contributed to the regulatory suspension and the subsequent repercussions faced by the company.
Duration temporary (a) The software failure incident in the articles appears to be temporary. The incident involved a suspension of a partnership between Alibaba Cloud and the Chinese telecommunications regulator due to Alibaba Cloud's failure to promptly report and address a cybersecurity vulnerability related to the Log4j software [122008, 122105]. The suspension is stated to be reassessed in six months, indicating that it is not a permanent ban but rather a temporary measure to be reviewed based on the company's internal reforms [122105].
Behaviour omission, timing, other (a) crash: The software failure incident in the articles does not involve a crash where the system loses state and does not perform any of its intended functions. [122008, 122105] (b) omission: The software failure incident is related to omission as Alibaba Cloud failed to promptly report vulnerabilities in the Apache Log4j2 software to the Chinese telecommunications regulator, leading to the suspension of their partnership. This omission of reporting the security flaw in time resulted in the regulatory action. [122008, 122105] (c) timing: The software failure incident can be attributed to timing as Alibaba Cloud did eventually find and report the security flaw in the Apache Log4j2 software to the Apache Software Foundation, but it was deemed that they did not report it to the Ministry of Industry and Information Technology (MIIT) in China promptly enough, leading to the suspension of their partnership. The timing of reporting the vulnerability was crucial in this case. [122008, 122105] (d) value: The software failure incident does not involve the system performing its intended functions incorrectly. [122008, 122105] (e) byzantine: The software failure incident does not exhibit the system behaving erroneously with inconsistent responses and interactions. [122008, 122105] (f) other: The other behavior in this software failure incident is related to the failure to effectively support the ministry's efforts to manage cybersecurity threats and vulnerabilities, as reported by China Daily. This failure to support the ministry's efforts was a key factor in the suspension of the partnership. [122008]

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence harm, property, non-human, theoretical_consequence (b) harm: People were physically harmed due to the software failure The software failure incident related to the Log4Shell security flaw in the widely used Log4j software had the potential to cause serious harm. The Chinese telecommunications regulator suspended a partnership with Alibaba Cloud because the company failed to promptly report the vulnerability in Apache Log4j2, which could lead to remote control of equipment and serious harms such as the theft of sensitive information and interruption of equipment services [122008, 122105].
Domain information, finance, government (a) The software failure incident involving the Log4Shell security flaw in the Apache Log4j software impacted the industry related to information technology and cybersecurity. The flaw was discovered by Alibaba Cloud, a subsidiary of Alibaba Group, which is a major player in the tech industry [Article 122008, Article 122105]. (h) The incident also has implications for the finance industry as cybersecurity vulnerabilities can lead to potential risks such as theft of sensitive information and interruption of services, which are critical concerns for financial institutions and transactions [Article 122105]. (l) The government sector is affected as well, as the Chinese Ministry of Industry and Information Technology (MIIT) suspended a partnership with Alibaba Cloud due to the company's failure to promptly report the Log4Shell security flaw, highlighting the government's focus on cybersecurity and national security in online infrastructure [Article 122008, Article 122105]. (m) The incident also has broader implications beyond the specific industries mentioned, as it underscores the importance of cybersecurity in the digital age and the need for prompt reporting and addressing of vulnerabilities to mitigate risks across various sectors [Article 122008, Article 122105].

Sources

Back to List