Published Date: 2020-12-08
| Postmortem Analysis | |
|---|---|
| Timeline | 1. The software failure incident involving SolarWinds happened between March and June of the year the incident was reported [Article 108811]. 2. The incident was traced back to October 2019 [Article 109675]. 3. The incident began as early as October 2019 but went undetected for months [Article 109675]. 4. The incident was reported in late December [Article 109675]. 5. The incident was reported in February [Article 112310]. 6. The incident was reported last week by FireEye [Article 116764]. |
| System | 1. SolarWinds Orion software [108542, 108793, 108805, 108811, 108850, 109675, 110984, 111143, 112310, 116764] |
| Responsible Organization | 1. An "outside nation-state" infiltrated SolarWinds' systems with malware, suspected to be the Russian government [Article 108542]. 2. Malware from a suspected second perpetrator compromised SolarWinds' product [Article 108573]. 3. Suspected Chinese hackers exploited a flaw in SolarWinds' software to break into U.S. government computers [Article 110984]. |
| Impacted Organization | 1. SolarWinds customers, including government agencies like the Centers for Disease Control and Prevention, the state department, and the justice department [108542, 108752, 108768] 2. FireEye, a cybersecurity firm [108460, 108850] 3. Microsoft and its customers [108547, 108768, 108846] 4. Various organizations in the US, Canada, Mexico, Belgium, Spain, the United Kingdom, Israel, and the United Arab Emirates [108547] 5. Around 50 organizations impacted by the hacking campaign [108810] 6. All five branches of the US military, the Pentagon, the State Department, and the Office of the President of the United States (users of SolarWinds' Orion platform) [108811] 7. More than 250 US federal agencies and private companies, including tech companies like Cisco Systems Inc., Intel Corp, and Nvidia Corp, accounting firm Deloitte, software company VMware Inc, electronics maker Belkin International Inc, the California Department of State Hospitals, and Kent State University [109675] 8. USDA customers affected by the SolarWinds Orion Code Compromise [110984] 9. Local and state governments, critical infrastructure entities, and other private sector organizations [112310] |
| Software Causes | 1. The software failure incident was caused by hackers gaining entry into networks through a tainted software update, specifically targeting SolarWinds Orion software [108542, 108793, 108805, 108807, 108844, 108850, 112310]. 2. The attackers managed to insert malicious code into a legitimate software update of SolarWinds Orion, leading to a supply chain attack affecting thousands of companies and government agencies [112310]. 3. The software vulnerability allowed the attackers to compromise systems and gain unauthorized access to sensitive information [111142]. 4. The incident highlighted the issue of software supply chain vulnerabilities, where obscure software packages can have hidden vulnerabilities that affect the security of networks [109659]. 5. The attackers exploited a software flaw in SolarWinds Orion network monitoring software to compromise up to 18,000 customers, including sensitive federal agencies [110984]. |
| Non-software Causes | 1. Lack of awareness among users and the software provider about the malicious nature of the software update [Article 108807] 2. Economic incentives prioritizing short-term profit over product security, leading to compromised software development practices [Article 109659] 3. Poor configurations and controls on the customer's part, allowing hackers to exploit vulnerabilities [Article 111142] 4. Blind trust in vendors and cloud services, giving them broad access to employee email and corporate networks [Article 116764] |
| Impacts | 1. The attackers were able to monitor internal emails at some of the top agencies in the US, potentially extracting information from many targets [108542]. 2. More than 40 of Microsoft's customers across eight countries were running the compromised software, with 80% of them in the United States [108768]. 3. SolarWinds estimated that about 18,000 customers had installed the compromised software, making them vulnerable to spy operations [108788]. 4. The intrusion posed a "grave" risk to government and private networks, as stated by the Department of Homeland Security's cybersecurity arm [108797]. 5. The attackers gained access to government and private networks by inserting malicious code in recent versions of SolarWinds' premier software product, Orion, affecting at least 24 organizations across the US [109675]. |
| Preventions | 1. Implementing stricter controls and verification processes in the software supply chain to ensure the integrity of software updates [108460, 108601]. 2. Enhancing trust management throughout the software supply chain to mitigate risks of malicious software updates [108601]. 3. Limiting the reach of outside software with far-reaching privileges and selecting firms with strong security practices for key administrative functions [108601]. 4. Developing mechanisms for evaluating and ensuring the trustworthiness of software updates applied by organizations [108601]. 5. Improving the understanding and management of trust in software supply chains, acknowledging the inherent challenges in achieving full trust [108601]. 6. Prioritizing product security over short-term profit considerations in software development and procurement decisions [109659]. 7. Strengthening configurations and controls on the customer's part to prevent exploitation of software flaws [111142]. 8. Enhancing security measures and controls to prevent insider threats and unauthorized access to critical software systems [114051]. |
| Fixes | 1. Improving trust in the software supply chain and managing trust throughout the software supply chain [108601] 2. Limiting the reach of outside software with far-reaching privileges and selecting firms with strong security practices for key administrative functions [108601] 3. Developing mechanisms for evaluating software updates and building trust in software supply chains [108601] 4. Recognizing systemic issues and understanding that no new technical fix or policy alone can solve the problem [108601] 5. Ensuring prompt installation of patches and upgrades to nullify vulnerabilities in software products [108839, 108841] 6. Enhancing configurations and controls on the customer's part to prevent software vulnerabilities [111142] | References | 1. Microsoft [108768, 108846, 110995, 111142] 2. SolarWinds [108850, 109659, 109675, 112310, 114051] 3. US government agencies [108542, 108547, 108850, 109675, 110984, 112310] 4. Department of Homeland Security [108846] 5. FireEye [108850, 111142] 6. CrowdStrike [111142] 7. Reuters [108841, 108850, 110984, 112310] 8. The New York Times [109659, 114051] 9. Wall Street Journal [109675, 112310] |
| Category | Option | Rationale |
|---|---|---|
| Recurring | one_organization, multiple_organization | (a) one_organization: A similar incident has happened before with SolarWinds. In the recent software failure incident, hackers gained access to SolarWinds Orion's software, allowing them access to all of its customers' networks [Article 108805]. This incident is reminiscent of a previous attack in 2017 known as NotPetya, where hackers gained access to victims by compromising tax software widely used in Ukraine [Article 108601]. (b) multiple_organization: The software failure incident involving SolarWinds has affected multiple organizations. At least 24 organizations across the US, including tech companies like Cisco Systems Inc., Intel Corp, and Nvidia Corp, were infected by the exploited software [Article 109675]. Additionally, the incident has impacted thousands of organizations and at least nine federal agencies, indicating a widespread impact [Article 112310]. |
| Phase (Design/Operation) | design, operation | (a) The software failure incident related to the development phases: - The incident involved hackers managing to access a system used by SolarWinds to put together updates for its Orion product, allowing them to insert malicious code into legitimate software updates, leading to a supply chain attack [Article 112310]. - The SolarWinds software updates were compromised, introducing malicious elements into the software, which were released between March and June [Article 108844]. - The incident highlighted the vulnerability of the software supply chain, where attackers can compromise widely used software to gain access to multiple targets [Article 108601]. (b) The software failure incident related to the operation phases: - The attackers exploited the trust in software updates, gaining insider access to organizations by modifying legitimate code in software updates [Article 108807]. - Organizations running an updated version of SolarWinds' Orion network management software had a "backdoor" installed in their systems by the attackers, potentially allowing further exploitation [Article 108850]. - The incident underscored the dangers posed by supply chain attacks, which are stealthy, hard to detect, and difficult to manage because end-users do not control the security of the products they are using [Article 109446]. |
| Boundary (Internal/External) | within_system, outside_system | (a) within_system: The software failure incident related to the SolarWinds breach was primarily due to contributing factors that originated from within the system. SolarWinds' Orion network management software was compromised by hackers who inserted a "backdoor" into the product, allowing them to gain access to thousands of customers' computer systems [Article 108850]. The flaw in the software was exploited by both suspected Chinese and Russian hacker groups, with the Russian hackers hiding a "back door" in Orion software updates sent to customers [Article 110984]. SolarWinds itself came under scrutiny for vulnerabilities in its software, which were coding errors within the Orion software running on victim systems [Article 112310]. (b) outside_system: The software failure incident also involved contributing factors that originated from outside the system. SolarWinds reported that an "outside nation-state" infiltrated its systems with malware, indicating that the breach was caused by external actors [Article 108542]. The breach highlighted the vulnerability of the software supply chain, where hackers could exploit the trust placed in software to penetrate networks [Article 108601]. Additionally, the hack was managed from servers inside the US, and early warning sensors placed by Cyber Command and the National Security Agency inside foreign networks to detect potential attacks failed, indicating external factors at play [Article 109675]. |
| Nature (Human/Non-human) | non-human_actions, human_actions | (a) The software failure incident occurring due to non-human actions: - The incident involved a highly sophisticated, targeted, and manual supply chain attack by an outside nation-state, indicating a failure introduced without human participation [Article 109446]. - Malicious elements were introduced into software updates for SolarWinds products, suggesting a failure due to contributing factors introduced without human participation [Article 108844]. (b) The software failure incident occurring due to human actions: - The incident highlighted the challenge of managing trust throughout the software supply chain and the consequences of that failure, indicating a failure due to contributing factors introduced by human actions [Article 108601]. - Suspected Chinese hackers exploited a flaw in SolarWinds software to break into U.S. government computers, showcasing a failure introduced by human actions [Article 110984]. |
| Dimension (Hardware/Software) | software | (a) The articles do not provide information about the software failure incident occurring due to contributing factors originating in hardware. (b) The software failure incidents reported in the articles are primarily due to contributing factors originating in software. For example, in the SolarWinds incident, hackers managed to access the system SolarWinds uses to put together updates for its Orion product and inserted malicious code into legitimate software updates, leading to a supply chain attack [Article 112310]. Additionally, the attackers exploited trusted software updates to gain access to thousands of companies and government agencies, highlighting a failure in managing trust throughout the software supply chain [Article 108460]. The incident involving SolarWinds Orion software was a supply chain attack where attackers compromised trusted code to distribute malware to customers without detection [Article 108751]. |
| Objective (Malicious/Non-malicious) | malicious, non-malicious | (a) Malicious: - The software failure incident was a result of a highly sophisticated, targeted, and manual supply chain attack by an outside nation-state, likely Moscow [Article 109446]. - Hackers inserted their own code into SolarWinds software through a supply chain attack, allowing them to gain access to sensitive systems without being detected [Article 108793]. - Russian hackers compromised software updates for SolarWinds Orion, enabling them to distribute malware to customers without detection [Article 108751]. (b) Non-malicious: - Users may not always download patches quickly, leaving themselves exposed to vulnerabilities in software products [Article 108790]. - The software supply chain is complex, making it almost impossible to comprehensively audit or understand, leading to vulnerabilities being exploited [Article 108601]. - The SolarWinds software updates were manipulated by hackers, potentially due to manual intervention, allowing for the introduction of malicious elements [Article 108844]. |
| Intent (Poor/Accidental Decisions) | poor_decisions, accidental_decisions | (a) The intent of the software failure incident related to poor decisions: - The SolarWinds breach was attributed to economic incentives that led to the company being breached and insecure software ending up in critical US government networks. The company ignored basic security practices and moved software development to Eastern Europe, potentially exposing it to subversion due to cost considerations [109659]. - The FireEye incident involved the disclosure of hacking tools, potentially making attackers' jobs easier by focusing on known software vulnerabilities [108839]. - The software supply chain attacks, such as the Russian operation targeting tax software in Ukraine and Chinese espionage operations compromising various companies, highlight the vulnerability of the software supply chain due to poor trust management and reliance on potentially compromised software [108601]. (b) The intent of the software failure incident related to accidental decisions: - The SolarWinds breach involved introducing malicious elements into software updates for SolarWinds products, potentially due to manual intervention requirements that could have been exploited by state actors [108844]. - The suspected Chinese group exploited a software flaw separate from the one used by Russian government operatives in the SolarWinds compromise, indicating separate groups of hackers targeting the same software product [110984]. - CrowdStrike faced a hacking attempt through a third-party vendor of Microsoft software, indicating a potential accidental decision in granting access to systems [111142]. |
| Capability (Incompetence/Accidental) | development_incompetence | (a) The software failure incident related to development incompetence is evident in the case of SolarWinds. The company ignored basic security practices, moved software development to Eastern Europe where there could be potential influence, and prioritized short-term profit over product security [Article 109659]. Additionally, SolarWinds has come under scrutiny for vulnerabilities in its software, which are coding errors not the result of attackers entering SolarWinds systems to implant malware [Article 112310]. (b) The accidental software failure incident is exemplified by the case where suspected Chinese hackers exploited a flaw in software made by SolarWinds to break into U.S. government computers. This was a new twist in a cybersecurity breach that U.S. lawmakers labeled a national security emergency, and separate from the Russian government's exploitation of a different software flaw in SolarWinds' Orion network monitoring software [Article 110984]. |
| Duration | permanent, temporary | (a) The software failure incident in the articles appears to be temporary rather than permanent. The incident involved a hacking campaign that lasted for months, allowing the hackers ample time to extract information from various targets [Article 108542]. The attack was described as ongoing, indicating that the breach was not a one-time event but a continuous operation [Article 108752]. Additionally, the incident highlighted the need for organizations to prioritize detection and response as much as prevention, suggesting a focus on addressing the immediate impact of the breach rather than accepting it as a permanent state [Article 108601]. (b) The incident also had elements of a permanent failure, as it was mentioned that the attackers had demonstrated patience, operational security, and complex tradecraft, indicating a sustained and ongoing effort to maintain access to the compromised networks [Article 108808]. The fact that the attack was described as having devastating and wide-ranging effects, with potential long-term consequences on private networks, critical infrastructure, and sensitive sectors, suggests that the impact of the breach could be long-lasting [Article 108751]. |
| Behaviour | crash, omission, timing, value, byzantine | (a) crash: Article 108788 reports a software failure incident where SolarWinds alerted its customers that an "outside nation state" had found a back door into its product, Orion, which helps monitor computer networks and servers. This incident can be categorized as a crash where the system lost its state and failed to perform its intended functions [108788]. (b) omission: The incident described in Article 109659 involving SolarWinds and the breach by Russia's SVR highlights a failure due to the system omitting to perform its intended functions. The article mentions that SolarWinds ignored basic security practices and moved software development to Eastern Europe, potentially leading to hidden vulnerabilities that affected the security of networks [109659]. (c) timing: The incident mentioned in Article 108460 discusses a software failure where attackers exploited trusted software updates to gain unauthorized access to thousands of companies and government agencies. This failure can be categorized as a timing issue where the system performed its intended functions correctly but at the wrong time, leading to security breaches [108460]. (d) value: The software failure incident reported in Article 110984 involves a software flaw exploited by a suspected Chinese group separate from the Russian government's exploitation of SolarWinds' Orion software. This incident can be categorized as a failure due to the system performing its intended functions incorrectly, leading to security breaches [110984]. (e) byzantine: The incident described in Article 110711 involving sophisticated techniques used by hackers to cover tracks and make forensic investigations difficult can be categorized as a byzantine failure. The system behaved erroneously with inconsistent responses and interactions, making it challenging to detect and analyze the attack [110711]. (f) other: The articles do not provide specific information on a failure behavior that does not fall under the categories of crash, omission, timing, value, or byzantine. |
| Layer | Option | Rationale |
|---|---|---|
| Perception | None | None |
| Communication | None | None |
| Application | None | None |
| Category | Option | Rationale |
|---|---|---|
| Consequence | property, non-human, theoretical_consequence, other | (a) death: People lost their lives due to the software failure - No information about people losing their lives due to the software failure was mentioned in the provided articles. (b) harm: People were physically harmed due to the software failure - No information about people being physically harmed due to the software failure was mentioned in the provided articles. (c) basic: People's access to food or shelter was impacted because of the software failure - No information about people's access to food or shelter being impacted due to the software failure was mentioned in the provided articles. (d) property: People's material goods, money, or data was impacted due to the software failure - The software failure incident resulted in damage to critical infrastructure, federal agencies, and private sector companies, posing a "grave threat" and potentially leading to the theft of information [Article 108808]. (e) delay: People had to postpone an activity due to the software failure - No information about people having to postpone activities due to the software failure was mentioned in the provided articles. (f) non-human: Non-human entities were impacted due to the software failure - The software failure incident impacted government agencies, private networks, and critical infrastructure [Article 108797, Article 108808]. (g) no_consequence: There were no real observed consequences of the software failure - The software failure incident had significant consequences, including a "grave" risk to government and private networks [Article 108797]. (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur - The articles discussed potential consequences such as the impact on national security, espionage, and the difficulty in managing supply chain attacks [Article 108601, Article 108751, Article 109446]. (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? - The software failure incident raised concerns about the vulnerability of government and private sector networks to attack and highlighted failures in the nation's cyberdefenses [Article 114051]. |
| Domain | information, government | (a) The failed system was related to the production and distribution of information, as it affected key federal agencies, tech and security companies, and the broader economy in terms of data access and sharing vulnerabilities [Article 108460, Article 108547, Article 108751, Article 108752, Article 109659, Article 109980, Article 114051]. (b) There is no specific mention of the transportation industry being directly impacted by the software failure incident. (c) There is no specific mention of the natural resources industry being directly impacted by the software failure incident. (d) There is no specific mention of the sales industry being directly impacted by the software failure incident. (e) There is no specific mention of the construction industry being directly impacted by the software failure incident. (f) There is no specific mention of the manufacturing industry being directly impacted by the software failure incident. (g) There is no specific mention of the utilities industry being directly impacted by the software failure incident. (h) There is no specific mention of the finance industry being directly impacted by the software failure incident. (i) There is no specific mention of the knowledge industry being directly impacted by the software failure incident. (j) There is no specific mention of the health industry being directly impacted by the software failure incident. (k) There is no specific mention of the entertainment industry being directly impacted by the software failure incident. (l) The failed system was related to the government sector, as key federal agencies, including those overseeing national security and nuclear weapons, were targeted in the attack [Article 108547, Article 108751, Article 108752, Article 109675, Article 109980, Article 114051]. (m) The software failure incident was not directly related to any other industry mentioned in the options. |
Article ID: 108751
Article ID: 110995
Article ID: 112310
Article ID: 110711
Article ID: 111142
Article ID: 110980
Article ID: 109675
Article ID: 110976
Article ID: 115570
Article ID: 111091
Article ID: 111143
Article ID: 109659
Article ID: 108811
Article ID: 108841
Article ID: 110984
Article ID: 116832
Article ID: 108752
Article ID: 108797
Article ID: 108460
Article ID: 108788
Article ID: 108844
Article ID: 109446
Article ID: 108601
Article ID: 109980
Article ID: 108808
Article ID: 116764
Article ID: 120057
Article ID: 108768
Article ID: 108793
Article ID: 108810
Article ID: 108790
Article ID: 108807
Article ID: 114051
Article ID: 108846
Article ID: 115678
Article ID: 108805
Article ID: 116359
Article ID: 108839
Article ID: 108547
Article ID: 108542
Article ID: 108573
Article ID: 108827
Article ID: 108850