Published Date: 2019-07-31
| Postmortem Analysis | |
|---|---|
| Timeline | 1. The software failure incident involving Cisco selling video surveillance software with known vulnerabilities to U.S. federal and state governments happened around 2008, as mentioned in Article 87036. 2. The incident was first identified by the whistle-blower, James Glenn, in 2008, and he reported it to Cisco at that time. Cisco acknowledged the flaws in the software in 2013, indicating that the incident occurred before that date. |
| System | 1. Cisco's video surveillance software system 2. Cisco's Video Surveillance Manager 3. Cisco's surveillance system architecture 4. Cisco's surveillance system software vulnerabilities 5. Cisco's Video Surveillance Manager software [Cisco Systems - Article 86788, Article 86950, Article 87036] |
| Responsible Organization | 1. Cisco Systems - The software failure incident was caused by Cisco Systems selling video surveillance software with known vulnerabilities to various government agencies and other organizations [86788, 86950, 87036]. |
| Impacted Organization | 1. Federal, state, and local government agencies, including Homeland Security, the Secret Service, the Army, the Navy, the Marines, the Air Force, and the Federal Emergency Management Agency [86788, 86950]. 2. Hospitals, airports, schools, state governments, and federal agencies [86950]. 3. U.S. federal and state governments, including the U.S. Army, Navy, Air Force, and Marine Corps [87036]. |
| Software Causes | 1. The software failure incident was caused by a significant security flaw in the video surveillance software sold by Cisco to various government agencies and other organizations [86788, 86950, 87036]. 2. The vulnerability in the software allowed hackers to gain unauthorized access to the video surveillance system, manipulate information, bypass security measures, turn surveillance cameras on and off, delete footage, and potentially compromise other connected physical security systems without being detected [86788, 86950, 87036]. 3. The flaw in the software was identified in 2008 by a whistle-blower, James Glenn, who discovered that he could hack into the video software and take over the surveillance system without being detected [86788, 86950, 87036]. 4. Cisco continued to sell the software with the vulnerability until 2013 when the company released software updates to fix the problems [86788, 86950, 87036]. 5. The software failure incident highlighted the lack of complete cybersecurity protections in the surveillance system, which required customers to add their own customized security tools on top of the software [86950]. 6. The software failure incident also raised concerns about the company's prioritization of profit and reputation over addressing the security flaws in the software [87036]. |
| Non-software Causes | 1. Lack of complete cybersecurity protections intentionally designed into the surveillance system by Cisco to allow customers to add their own security tools [86950]. 2. Failure to address the security weakness in the software for about four years after it was first reported by the whistle-blower in 2008 [86950]. 3. Prioritization of profit and reputation over addressing the security vulnerability in the software [87036]. |
| Impacts | 1. The software failure incident involving Cisco's video surveillance technology had significant security flaws that rendered the software "of no value" to government agencies, including Homeland Security, the Secret Service, and various military branches [86788, 86950]. 2. Hackers could exploit the vulnerability in the software to gain unauthorized access to surveillance systems, manipulate information, and bypass security measures without being detected [86788, 86950]. 3. The flaw in the software allowed hackers to potentially turn surveillance cameras on and off, delete footage, and compromise other connected physical security systems such as alarms or locks [86950]. 4. The software vulnerability was not addressed by Cisco for about four years after it was first reported by a whistleblower in 2008, leading to a prolonged period of exposure to potential security breaches [86950]. 5. The incident marked the first time a company was forced to pay out under a federal whistleblower law for failing to have adequate cybersecurity protections, highlighting the legal and financial consequences of software failures in terms of cybersecurity [86950]. 6. The settlement amount paid by Cisco to resolve the claim was $8.6 million, with most of the funds going to the federal government and 15 state buyers, and over $1 million awarded to the whistleblower, James Glenn [87036]. 7. The incident raised concerns about the pervasiveness of weak software in critical sectors such as hospitals, airports, schools, and government agencies, emphasizing the urgent need for robust cybersecurity measures in software development and deployment [86950]. 8. The case highlighted the importance of ethical considerations in the tech community, with the whistleblower expressing hope that his experience would prompt others to prioritize ethical mandates over profit and reputation [87036]. |
| Preventions | 1. Timely Response to Vulnerability Reports: The software failure incident could have been prevented if Cisco had promptly addressed the security flaw reported by the whistle-blower in 2008, rather than waiting for several years to release a fix [86788, 86950, 87036]. 2. Regular Security Audits and Testing: Conducting regular security audits and testing of the software could have helped identify and rectify vulnerabilities before they could be exploited by hackers [86788, 86950, 87036]. 3. Stronger Cybersecurity Protocols: Implementing robust cybersecurity protocols within the software itself could have prevented unauthorized access and manipulation of the surveillance system by hackers [86788, 86950, 87036]. 4. Ethical Considerations: Prioritizing ethical considerations over profit and reputation, as mentioned by the whistle-blower, could have led to a more proactive approach in addressing security flaws and ensuring the software's integrity [87036]. |
| Fixes | 1. Implementing software updates and patches to fix the vulnerabilities in the video surveillance software sold by Cisco [86788, 86950, 87036]. 2. Enhancing cybersecurity measures and ensuring that the software meets industry standards for security [86788, 86950, 87036]. 3. Conducting thorough security testing and audits to identify and address any potential flaws or weaknesses in the software [86788, 86950, 87036]. 4. Encouraging a culture of prioritizing ethical considerations and doing what is right over profit and reputation in the tech community [87036]. | References | 1. James Glenn, the whistle-blower who discovered the software vulnerability and reported it to Cisco [Article 86788, Article 86950, Article 87036] 2. Constantine Cannon, the law firm representing James Glenn [Article 86788, Article 86950, Article 87036] 3. Cisco Systems Inc. [Article 86788, Article 86950, Article 87036] 4. U.S. federal and state governments [Article 86788, Article 86950, Article 87036] 5. Justice Department [Article 86788] 6. Federal Trade Commission (FTC) [Article 86788] 7. Mark Chandler, Cisco's Chief Legal Officer [Article 86950] 8. Anne Hayes Hartman, attorney for James Glenn [Article 87036] 9. Georgetown University law professor Gregory Klass [Article 87036] 10. L.A. airport police detective [Article 87036] |
| Category | Option | Rationale |
|---|---|---|
| Recurring | one_organization, multiple_organization | (a) The software failure incident having happened again at one_organization: - The software vulnerability in Cisco's video surveillance software was identified in 2008 by a whistle-blower, James Glenn, who discovered that he could hack into the video software and take over the surveillance system without being detected [Article 86788]. - Cisco continued to sell the software with the vulnerability until July 2013 when the company released a way to fix the problem [Article 86788]. - Cisco's Chief Legal Officer mentioned that the surveillance system was designed without complete cybersecurity protections, and customers were advised to add their own security tools on top of the software [Article 86950]. (b) The software failure incident having happened again at multiple_organization: - The flawed Cisco software was sold to hospitals, airports, schools, state governments, and federal agencies [Article 86950]. - The complaint also mentioned that the U.S. Army, Navy, Air Force, and Marine Corps were customers of the vulnerable video surveillance software [Article 87036]. - Cisco's Video Surveillance Manager was used by various entities such as Los Angeles International Airport, the Washington D.C. police, the New York City public transit system, and many schools [Article 87036]. |
| Phase (Design/Operation) | design, operation | (a) The software failure incident in the articles was primarily related to the design phase. Cisco Systems sold video surveillance software with a significant security flaw to various government agencies, hospitals, airports, schools, and other entities. The flaw allowed hackers to exploit the software, potentially compromising surveillance cameras and connected physical security systems without being detected. The vulnerability was identified in 2008 by a whistle-blower, but Cisco continued to sell the software with the vulnerability until 2013 when they released software updates to fix the problems [86788, 86950, 87036]. (b) The software failure incident also involved operational factors. The flaw in the video surveillance software could be exploited by hackers to turn surveillance cameras on and off, delete footage, and potentially compromise other connected physical security systems like alarms or locks. Despite the vulnerability, there was no evidence that unauthorized access to customers' video occurred as a result of the architecture. The settlement highlighted the importance of cybersecurity standards in government contracts and the need for vendors to address vulnerabilities promptly to ensure operational security [86788, 86950, 87036]. |
| Boundary (Internal/External) | within_system, outside_system | (a) within_system: - The software failure incident involving Cisco's video surveillance technology was due to a significant security flaw within the system itself [86788, 86950, 87036]. - The vulnerability in the software allowed hackers to gain unauthorized access to the video surveillance system, manipulate information, and bypass security measures [86788, 86950, 87036]. - The flaw was identified by a whistle-blower who discovered that he could hack into the video software and take over the surveillance system without being detected [86788, 86950, 87036]. - Cisco continued to sell the software with the vulnerability until 2013 when they released software updates to fix the problems [86788, 86950, 87036]. (b) outside_system: - The software failure incident was also influenced by external factors such as the lack of cybersecurity standards and oversight in government contracts [86950, 87036]. - The case was filed under the False Claims Act, which addresses fraud and misconduct in federal government contracts, indicating a failure in the oversight and enforcement of cybersecurity standards [86788, 86950, 87036]. - The vulnerability in the software allowed hackers to potentially compromise other connected physical security systems, indicating a broader impact beyond the software itself [86950]. - The settlement highlighted the need for stronger cybersecurity practices and the potential risks associated with hackable products purchased by federal agencies and other entities [86950]. |
| Nature (Human/Non-human) | non-human_actions, human_actions | (a) The software failure incident in the articles was primarily due to non-human actions, specifically a significant security flaw in the video surveillance software sold by Cisco Systems. This flaw allowed hackers to gain unauthorized access to the surveillance system, manipulate information, and bypass security measures without being detected [86788, 86950, 87036]. (b) Human actions also played a role in the software failure incident. The whistleblower, James Glenn, who was working as a Cisco subcontractor in Denmark, discovered the vulnerability in the software in 2008 and reported it to Cisco. However, Cisco did not address the flaw for several years despite being aware of it, leading to the continued sale of the software with the vulnerability [86788, 86950, 87036]. |
| Dimension (Hardware/Software) | software | (a) The articles do not provide information about the software failure incident occurring due to contributing factors originating in hardware. (b) The software failure incident reported in the articles was due to contributing factors that originated in software. Cisco Systems sold video surveillance software with a significant security flaw that allowed hackers to gain unauthorized access to the surveillance system, manipulate information, and bypass security measures [86788, 86950, 87036]. The vulnerability in the software was identified by a whistle-blower in 2008, who discovered that he could hack into the video software and take over the surveillance system without being detected [86788]. Despite being made aware of the flaw, Cisco continued to sell the software with the vulnerability until 2013 when they released software updates to fix the problems [86788]. The flaw in the software allowed hackers to spy on video footage, turn surveillance cameras on and off, delete footage, and potentially compromise other connected physical security systems [86950]. Cisco's Chief Legal Officer mentioned that the surveillance system was designed without complete cybersecurity protections, and customers were advised to add their own security tools on top of the software [86950]. The settlement with the Justice Department and states marked the first time a company was forced to pay out under a federal whistleblower law for failing to have adequate cybersecurity protections [86950]. |
| Objective (Malicious/Non-malicious) | non-malicious | (a) The software failure incident in the articles was non-malicious. The incident involved a significant security flaw in Cisco's video surveillance software that allowed hackers to exploit vulnerabilities without being detected. The flaw was identified by a whistle-blower in 2008, and despite being reported to Cisco, the company did not address the issue promptly. The vulnerability allowed unauthorized access to surveillance systems, manipulation of information, and bypassing security measures [86788, 86950, 87036]. |
| Intent (Poor/Accidental Decisions) | poor_decisions | (a) The intent of the software failure incident was due to poor_decisions. The incident involved Cisco selling video surveillance software with known vulnerabilities to various government agencies, hospitals, airports, schools, and other entities. Despite being alerted about the security flaw in 2008, Cisco continued to sell the software without fixing the issue for about four years [86788, 86950, 87036]. Cisco's Chief Legal Officer mentioned that the surveillance system was purposely designed without complete cybersecurity protections so that customers could add their own customized security tools. The company only alerted customers in 2009 that they needed to pay special attention to building necessary security features on top of the software, and it was not until 2013 that Cisco concluded that customers needed to upgrade to a newer version with fuller digital protections [86950]. |
| Capability (Incompetence/Accidental) | development_incompetence | (a) The software failure incident in the articles appears to be related to development incompetence. Cisco Systems sold video surveillance software with a significant security flaw to various government agencies, hospitals, airports, schools, and federal agencies. The flaw allowed hackers to potentially compromise surveillance cameras, turn them on and off, delete footage, and even compromise other connected physical security systems without being detected [Article 86788, Article 86950, Article 87036]. The vulnerability was identified by a whistle-blower in 2008, but Cisco continued to sell the software with the vulnerability until 2013, when they released software updates to fix the problems. This incident highlights a failure in professional competence by Cisco in ensuring the security of their software products. |
| Duration | temporary | The software failure incident related to the Cisco video surveillance software vulnerability can be categorized as a temporary failure. The vulnerability in the software was identified in 2008 by a whistle-blower, James Glenn, who discovered that he could hack into the video software and take over the surveillance system without being detected [Article 86788]. Despite Glenn's report in 2008, Cisco did not fix the flaw until July 2013, when they released software updates to address the vulnerabilities [Article 86788]. This indicates that the software failure was temporary, lasting for about five years until a fix was implemented. |
| Behaviour | value, other | (a) crash: The software failure incident did not involve a crash where the system loses state and does not perform any of its intended functions [86788, 86950, 87036]. (b) omission: The failure was not due to the system omitting to perform its intended functions at an instance(s) [86788, 86950, 87036]. (c) timing: The failure was not due to the system performing its intended functions correctly, but too late or too early [86788, 86950, 87036]. (d) value: The software failure incident involved a failure where the system performed its intended functions incorrectly, as the video surveillance software sold by Cisco had a significant security flaw that reduced the protection provided by other security systems [86788, 86950, 87036]. (e) byzantine: The failure was not due to the system behaving erroneously with inconsistent responses and interactions [86788, 86950, 87036]. (f) other: The behavior of the software failure incident was related to a significant security flaw in the video surveillance software sold by Cisco, which allowed hackers to potentially compromise surveillance cameras, turn them on and off, delete footage, and compromise other connected physical security systems without being detected [86788, 86950, 87036]. |
| Layer | Option | Rationale |
|---|---|---|
| Perception | None | None |
| Communication | None | None |
| Application | None | None |
| Category | Option | Rationale |
|---|---|---|
| Consequence | property, non-human, theoretical_consequence | The consequence of the software failure incident described in the articles is primarily related to potential harm and property impact due to the security flaw in the video surveillance software sold by Cisco Systems. The vulnerability in the software could have allowed hackers to compromise surveillance cameras, potentially leading to unauthorized access, manipulation of information, and bypassing security measures without being detected. However, there is no evidence that the vulnerability was actually exploited to spy on any Cisco customers' cameras. The software flaw could have enabled hackers to turn surveillance cameras on and off, delete footage, and compromise other connected physical security systems such as alarms or locks [Article 86950]. Additionally, the software failure incident resulted in a financial consequence for Cisco Systems, as they agreed to pay $8.6 million to settle the claim related to selling the vulnerable video surveillance software to various government agencies and entities [Article 86788, Article 86950, Article 87036]. |
| Domain | government | The failed system in the reported software failure incident was related to the government industry. The software was sold to federal, state, and local government agencies, including Homeland Security, the Secret Service, the Army, the Navy, the Marines, the Air Force, the Federal Emergency Management Agency, prisons, police departments, and government grant recipients such as schools and hospitals [86788, 86950, 87036]. |
Article ID: 86788
Article ID: 86950
Article ID: 87036