Incident: Vulnerabilities in VxWorks Networking Protocols Impact Critical Systems

Published Date: 2019-07-29

Postmortem Analysis
Timeline 1. The software failure incident happened in early August 2019 [89480]. 2. The vulnerabilities in the VxWorks operating system were disclosed in March 2019 [87289].
System 1. VxWorks operating system [89480, 87289] 2. IPnet networking code 3. Real-time operating systems (RTO systems) 4. Devices running VxWorks, ENEA's Operating System Embedded, and other affected operating systems
Responsible Organization 1. Armis researchers [89480, 87289] 2. Wind River [89480, 87289] 3. Interpeak [89480] 4. ENEA [89480] 5. Department of Homeland Security [89480] 6. Food and Drug Administration [89480] 7. Real-time operating system and device companies [89480]
Impacted Organization 1. The Department of Homeland Security [89480] 2. The Food and Drug Administration [89480] 3. Various real-time operating system and device companies [89480] 4. Medical device manufacturer Becton Dickinson Alaris [89480] 5. Device makers and manufacturers of medical and industrial devices [89480] 6. Users of devices with vulnerable operating systems [87289] 7. Organizations deploying devices with VxWorks [87289]
Software Causes 1. The software causes of the failure incident were a suite of network protocol bugs known as Urgent/11 that exist in various real-time operating systems, including VxWorks, ENEA's Operating System Embedded, and others [89480]. 2. The vulnerabilities in the network protocols of VxWorks, specifically in the TCP/IP stack, allowed for remote device access and potential malware spread, affecting roughly 200 million vulnerable devices [87289].
Non-software Causes 1. Lack of standardization in software components and open-source modules used in various software products [89480] 2. Challenges in applying security patches due to the lack of resources or time to update IoT devices [89480] 3. Difficulty in implementing fixes for continuously running devices like patient monitors and medical equipment [87289]
Impacts 1. The software failure incident involving the Urgent/11 vulnerabilities in VxWorks had disturbing implications for the security of critical systems such as patient monitors, routers, security cameras, and more across various industries [89480]. 2. The vulnerabilities could potentially allow attackers to gain remote device access and spread malware to vulnerable devices, impacting roughly 200 million devices worldwide [87289]. 3. The vulnerabilities could lead to device malfunctions or even full system takedowns, similar to the impact of the WannaCry ransomware on Windows machines [87289]. 4. The vulnerabilities in the network protocol layer could be exploited from afar without needing a foothold from the victim, posing a significant threat to industrial devices and critical infrastructure [87289].
Preventions 1. Regular security audits and vulnerability assessments could have potentially prevented the software failure incident by identifying and addressing the vulnerabilities in the networking protocols of the operating systems [87289]. 2. Implementing a robust patch management process that ensures timely distribution and installation of security patches could have helped mitigate the risks associated with the vulnerabilities in the affected operating systems [87289]. 3. Adoption of a "software bill of materials" by manufacturers, as advocated by the FDA, could have facilitated tracking vulnerabilities across various devices and software components, making it easier to address issues like the ones found in the incident [89480]. 4. Enhanced collaboration between security researchers, manufacturers, regulators, and industry stakeholders, as demonstrated at events like Defcon's Biohacking Village, could have led to earlier detection and mitigation of the vulnerabilities in critical systems [89480]. 5. Improved industry-wide standardization and security practices for embedded devices and IoT systems could help prevent similar incidents in the future by ensuring that security vulnerabilities are addressed proactively [87289, 89480].
Fixes 1. Patch distribution for vulnerable devices by Wind River to address the Urgent/11 vulnerabilities in VxWorks [87289]. 2. Implementation of mitigation techniques by BD Alaris, such as specific firewall rules, to block remote attempts to exploit the IPnet bugs in their products [89480]. 3. Adoption of a "software bill of materials" by manufacturers, as advocated by the FDA, to outline components in devices for easier tracking of vulnerabilities like Urgent/11 [89480].
References 1. Armis [89480, 87289] 2. Department of Homeland Security [89480] 3. Food and Drug Administration [89480] 4. Wind River [89480, 87289] 5. Becton Dickinson Alaris [89480] 6. ENEA [89480] 7. Interpeak [89480] 8. Microsoft [87289] 9. Mentor (owned by Siemens) [89480] 10. Cybersecurity and Infrastructure Security Agency [89480]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) In the provided articles, the software failure incident related to vulnerabilities in the VxWorks operating system has happened again at the same organization, Wind River. The vulnerabilities, collectively dubbed Urgent/11, were found in VxWorks, impacting various devices. Wind River, the developer of VxWorks, is in the process of distributing patches for the bugs [87289]. (b) The software failure incident related to vulnerabilities in the VxWorks operating system has also happened at multiple organizations. The vulnerabilities in the networking protocols of VxWorks could potentially affect roughly 200 million devices worldwide, impacting various industries such as medical equipment, industrial control products, firewalls, routers, printers, and more. The vulnerabilities could give attackers remote device access and allow malware to spread to other VxWorks devices globally [87289].
Phase (Design/Operation) design, operation (a) In the software failure incident related to the vulnerabilities in the VxWorks operating system, the failure can be attributed to contributing factors introduced during the design phase of the system development. The vulnerabilities in the networking protocols of VxWorks were discovered by Armis researchers, highlighting flaws in the system's design that allowed for remote device access and potential malware spread [87289]. (b) Additionally, the software failure incident can also be linked to contributing factors introduced during the operation phase of the system. The challenges in patching the vulnerabilities in VxWorks devices were emphasized, with the continuous operation of devices and the need for tailored patching processes making it difficult to implement fixes promptly [87289].
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident related to the vulnerabilities in the VxWorks operating system, known as Urgent/11, was primarily within the system. Armis researchers discovered a cluster of 11 vulnerabilities in the platform's networking protocols, which could give an attacker remote device access and allow malware to spread to other VxWorks devices [87289]. These vulnerabilities were found in the TCP/IP stack, a foundational component that helps devices connect to networks like the internet [87289]. The vulnerabilities were present in most versions of VxWorks going back to 2006, affecting roughly 200 million devices [87289]. (b) outside_system: The software failure incident also had contributing factors originating from outside the system. The vulnerabilities in VxWorks were not limited to just that operating system but extended to other platforms that incorporated the same decades-old networking code, such as IPnet, which was licensed to various real-time operating system developers [89480]. The acquisition of Interpeak by Wind River and the subsequent lack of support for IPnet licenses allowed the vulnerabilities to persist across different systems [89480]. Additionally, the vulnerabilities could be exploited remotely, without needing a foothold from a victim, making them susceptible to attacks from afar [87289].
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident occurring due to non-human actions: - The software failure incident involving vulnerabilities in the VxWorks operating system, known as Urgent/11, was due to a suite of network protocol bugs that existed in various platforms across different manufacturers [89480]. - The vulnerabilities in the VxWorks operating system's networking protocols were discovered by researchers from Armis, and these vulnerabilities could potentially give an attacker remote device access and allow malware to spread to other VxWorks devices worldwide [87289]. - The vulnerabilities in the VxWorks operating system's network protocols were unusual and had a broad reach, affecting devices like modems, routers, printers, industrial, and medical devices [87289]. (b) The software failure incident occurring due to human actions: - The vulnerabilities in the VxWorks operating system were found by researchers from Armis, who disclosed their findings to Wind River in March, initiating the patching process [87289]. - Wind River, the developer of VxWorks, has been working with customers to distribute patches for the vulnerabilities, but the patching process is challenging due to the continuous operation of VxWorks devices and the need for tailored patching processes [87289]. - The challenges in patching VxWorks devices highlight the difficulty in mitigating the fallout of vulnerabilities introduced in the software due to the nature of IoT and critical infrastructure updates [87289].
Dimension (Hardware/Software) hardware, software (a) The software failure incident occurring due to hardware: - The software failure incident reported in the articles is related to vulnerabilities found in the networking protocols of the VxWorks operating system, which is commonly used in devices like medical equipment, elevator controllers, and satellite modems [87289]. - The vulnerabilities in the VxWorks operating system's network protocols could potentially give attackers remote device access and allow malware to spread to other VxWorks devices worldwide, impacting roughly 200 million vulnerable devices [87289]. - The vulnerabilities in the VxWorks operating system's network protocols are considered critical and have a broad reach, affecting devices such as modems, routers, printers, industrial devices, and medical devices [87289]. - The nature of VxWorks devices, which often run continuously and depend on customized software, makes it challenging to implement patches for the vulnerabilities, as shutting down a product line to apply updates is not a feasible solution [87289]. (b) The software failure incident occurring due to software: - The software failure incident is primarily attributed to vulnerabilities found in the networking protocols of the VxWorks operating system, which is a software component used in various devices [87289]. - The vulnerabilities in the VxWorks operating system's network protocols were discovered by researchers from Armis, highlighting a cluster of 11 vulnerabilities, six of which could potentially allow remote device access and malware propagation [87289]. - The vulnerabilities in the VxWorks operating system's network protocols are considered critical and could result in device malfunctions or full system takedowns if exploited by attackers [87289]. - Wind River, the developer of VxWorks, is in the process of distributing patches for the vulnerabilities, emphasizing the importance of patching impacted devices immediately to mitigate the risks posed by the software vulnerabilities [87289].
Objective (Malicious/Non-malicious) malicious (a) The software failure incident discussed in the articles is malicious in nature. The vulnerabilities found in the VxWorks operating system, collectively dubbed Urgent/11, were discovered by researchers from Armis and could potentially give an attacker remote device access and allow malware to spread to other VxWorks devices [87289]. These vulnerabilities could be exploited by motivated attackers to launch attacks at a massive scale, posing a significant threat to critical infrastructure and industrial devices [87289]. The presence of these vulnerabilities in the network protocols of VxWorks allows for remote exploitation without needing a foothold from the victim, making it a serious security concern [87289]. Additionally, the incident involved a suite of network protocol bugs, Urgent/11, that exist in various platforms used in always-on devices common to industries like industrial control and healthcare [89480]. The bugs were found to have endured for a long time due to the lack of support for the IPnet licenses, leaving devices vulnerable to denial of service attacks or full takeovers [89480]. The vulnerabilities discovered in the operating systems of devices like patient monitors, routers, security cameras, and infusion pumps highlight the potential for malicious exploitation of these systems [89480].
Intent (Poor/Accidental Decisions) poor_decisions (a) The intent of the software failure incident related to poor decisions can be seen in the software failure incident reported in Article 89480. The incident involved vulnerabilities in the Urgent/11 suite of network protocol bugs that affected various real-time operating systems used in critical systems like patient monitors, routers, and security cameras. The vulnerabilities stemmed from a popular early-aughts implementation of network protocols that were inherited through acquisitions and lack of support for older software components. This lack of management and oversight of embedded devices led to the persistence of vulnerabilities over time, despite significant code changes in the systems. The incident highlights the consequences of poor decisions in managing software components and the challenges in addressing vulnerabilities in legacy systems ([89480]). (b) The software failure incident related to accidental decisions can be observed in the vulnerabilities discovered in the VxWorks operating system, as reported in Article 87289. The vulnerabilities, collectively known as Urgent/11, were found in the networking protocols of VxWorks, a real-time operating system widely used in IoT and industrial control products. The presence of these vulnerabilities in the network layer was unexpected and unusual, as security vulnerabilities in this foundational component had been largely standardized industry-wide since the 1990s. The discovery of critical vulnerabilities in VxWorks, a system known for its security, was surprising and highlighted the accidental nature of the flaws that could potentially impact millions of devices worldwide ([87289]).
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident in the articles can be attributed to development incompetence. The vulnerabilities in the VxWorks operating system, collectively known as Urgent/11, were discovered by Armis researchers and found to be present in most versions of VxWorks going back to 2006 [87289]. The vulnerabilities in the network protocols of VxWorks were considered critical, and the patching process was described as long and difficult, especially in the context of IoT and critical infrastructure updates [87289]. The challenges in distributing patches for VxWorks devices, which are used in various critical settings like patient monitors and medical equipment, were highlighted, indicating a lack of professional competence in managing and updating these systems effectively [87289]. (b) The software failure incident can also be considered accidental to some extent. The vulnerabilities in the VxWorks operating system were surprising due to their presence in the network protocols, particularly the TCP/IP stack, which is a foundational component for devices to connect to networks like the internet [87289]. The vulnerabilities were not expected in VxWorks, a secure and real-time operating system commonly used in continuously functioning devices like medical equipment and industrial control products [87289]. The accidental nature of these vulnerabilities is further emphasized by the fact that they were not present in the latest version of VxWorks or in Wind River's certification versions, indicating that they were not intentionally introduced but rather remained unnoticed until discovered by researchers [87289].
Duration permanent, temporary (a) The software failure incident related to the vulnerabilities in the VxWorks operating system, known as Urgent/11, can be considered as a permanent failure. The vulnerabilities discovered in the networking protocols of VxWorks have been present in most versions of the operating system going back to 2006 [87289]. The nature of VxWorks devices, which typically run continuously and often require customized software that complicates the patching process, makes it challenging to implement fixes for the vulnerabilities [87289]. Additionally, previous research on exploitable VxWorks bugs has shown that these vulnerabilities can persist for a long time in critical infrastructure devices, leading to what is referred to as "forever-day vulnerabilities" [87289]. (b) The software failure incident can also be considered as a temporary failure in the sense that patches are being distributed by the developer, Wind River, to address the Urgent/11 vulnerabilities in VxWorks. Wind River has been working with customers to distribute patches for the vulnerabilities for almost two months [87289]. However, the patching process is described as long and difficult, especially for IoT and critical infrastructure updates, indicating that the vulnerabilities are not immediately resolved for all affected devices [87289].
Behaviour omission, value, other (a) crash: The articles do not mention a crash as the behavior of the software failure incident. (b) omission: The software failure incident related to the vulnerabilities in VxWorks and the Urgent/11 bugs could lead to the omission of performing intended functions by the affected devices. For example, the vulnerabilities could potentially allow attackers to disrupt the normal operation of devices like patient monitors, routers, security cameras, and infusion pumps [89480, 87289]. (c) timing: The software failure incident does not relate to timing issues where the system performs its intended functions but at incorrect times. (d) value: The vulnerabilities in VxWorks and the Urgent/11 bugs could lead to the system performing its intended functions incorrectly, potentially allowing unauthorized access, manipulation of devices, and spreading of malware [89480, 87289]. (e) byzantine: The software failure incident does not exhibit a byzantine behavior where the system behaves erroneously with inconsistent responses and interactions. (f) other: The other behavior observed in the software failure incident is the potential for the vulnerabilities to allow remote device access, spreading of malware, and the possibility of full system takedowns, impacting critical infrastructure and devices [87289].

IoT System Layer

Layer Option Rationale
Perception network_communication, embedded_software (a) sensor: The software failure incident was related to vulnerabilities in the network protocols of the VxWorks operating system, affecting devices like medical equipment, elevator controllers, and satellite modems. These vulnerabilities could give attackers remote device access and allow malware to spread to other VxWorks devices [87289]. (b) actuator: The failure was not directly related to actuator errors but rather to vulnerabilities in the network protocols of the VxWorks operating system [87289]. (c) processing_unit: The failure was not directly related to processing errors but rather to vulnerabilities in the network protocols of the VxWorks operating system [87289]. (d) network_communication: The software failure incident was specifically related to vulnerabilities in the network protocols of the VxWorks operating system, which could allow attackers to exploit devices with networking capabilities [87289, 89480]. (e) embedded_software: The failure was related to vulnerabilities in the embedded software of various real-time operating systems, including VxWorks, ENEA's Operating System Embedded, and others, due to the presence of bugs in the TCP/IP stack that were inherited from the IPnet codebase [89480].
Communication connectivity_level [a87289, a89480] - The software failure incident related to the communication layer of the cyber-physical system that failed was at the connectivity_level. This is evident from the articles discussing vulnerabilities in the networking protocols, specifically the TCP/IP stack, which help devices connect to networks like the internet. The vulnerabilities found in the VxWorks operating system were related to the network layer, allowing attackers remote device access and the potential spread of malware to other devices [87289]. Additionally, the vulnerabilities were described as affecting devices that have networking capabilities, emphasizing the impact on the network or transport layer of the cyber-physical system [89480].
Application FALSE The software failure incident described in the articles was not related to the application layer of the cyber physical system that failed due to contributing factors introduced by bugs, operating system errors, unhandled exceptions, and incorrect usage. The failure was specifically related to vulnerabilities in the networking protocols of the VxWorks operating system, which affected various devices running on this platform [87289, 89480].

Other Details

Category Option Rationale
Consequence property, non-human, theoretical_consequence, other (a) death: People lost their lives due to the software failure - There is no mention of people losing their lives due to the software failure incident in the articles. (b) harm: People were physically harmed due to the software failure - The articles do not mention people being physically harmed due to the software failure incident. (c) basic: People's access to food or shelter was impacted because of the software failure - The articles do not mention people's access to food or shelter being impacted due to the software failure incident. (d) property: People's material goods, money, or data was impacted due to the software failure - The software failure incident impacted various devices such as infusion pumps, patient monitors, cameras, printers, routers, Wi-Fi mesh access points, and a Panasonic doorbell camera, making them vulnerable to exploitation [89480]. - The vulnerabilities in the software could potentially lead to device malfunctions or full system takedowns, affecting the functionality of these devices [87289]. (e) delay: People had to postpone an activity due to the software failure - The articles do not mention people having to postpone activities due to the software failure incident. (f) non-human: Non-human entities were impacted due to the software failure - The software failure incident impacted various IoT devices, including infusion pumps, patient monitors, cameras, printers, routers, and more, making them vulnerable to exploitation [89480]. - The vulnerabilities in the software could potentially lead to device malfunctions or full system takedowns, affecting the functionality of these devices [87289]. (g) no_consequence: There were no real observed consequences of the software failure - The software failure incident had real observed consequences, such as vulnerabilities in various devices and the potential for exploitation [89480, 87289]. (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur - The articles discuss potential consequences of the software failure, such as the possibility of device malfunctions, full system takedowns, and the need for patching vulnerable devices [89480, 87289]. (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? - The software failure incident led to the discovery of vulnerabilities in critical systems like patient monitors, routers, security cameras, and more, across various manufacturers, highlighting the security risks posed by unmanaged embedded devices [89480]. - The patching process for vulnerable devices was described as long and difficult, especially in the case of IoT and critical infrastructure updates, due to the continuous operation of devices and the need for tailored patching processes [87289].
Domain information, health (a) The failed system was intended to support the information industry as it was related to the production and distribution of information. The software failure incident involved vulnerabilities in a popular operating system called VxWorks, which is used in devices like medical equipment, elevator controllers, and satellite modems [Article 87289]. (b) The transportation industry was not directly mentioned in the articles. (c) The failed system was not directly related to the natural resources industry. (d) The failed system was not directly related to the sales industry. (e) The failed system was not directly related to the construction industry. (f) The failed system was not directly related to the manufacturing industry. (g) The failed system was not directly related to the utilities industry. (h) The failed system was not directly related to the finance industry. (i) The failed system was not directly related to the knowledge industry. (j) The failed system was intended to support the health industry as it involved vulnerabilities in medical devices like infusion pumps, patient monitors, and cameras [Article 89480]. (k) The failed system was not directly related to the entertainment industry. (l) The failed system was not directly related to the government industry. (m) The failed system was not directly related to any other specific industry mentioned in the articles.

Sources

Back to List