Published Date: 2015-07-08
| Postmortem Analysis | |
|---|---|
| Timeline | 1. The software failure incident at the New York Stock Exchange occurred on July 8, 2015 [38213, 38171, 37661]. |
| System | 1. Configuration problem in the system [38213, 38213] 2. Software update error [38213] 3. Internal technical issue [38171] 4. Software bug in the system [37661] 5. Configuration issue [37661] |
| Responsible Organization | 1. The software failure incident at the New York Stock Exchange was caused by an internal technical issue [Article 38171]. 2. The incident was also attributed to a configuration problem in the system that needed fixing [Article 38213]. |
| Impacted Organization | 1. The New York Stock Exchange (NYSE) [38213, 38171, 37661] |
| Software Causes | 1. A technical "configuration problem" that needed fixing, possibly related to a system update, leading to the NYSE shutdown [38213]. 2. An internal technical issue at the NYSE, not due to a cyberattack, causing the trading halt [38171]. 3. A bug in the software that was being updated before the trading day began, which could not be fixed without shutting down the system, leading to the NYSE shutdown [37661]. |
| Non-software Causes | 1. The outage at the New York Stock Exchange was caused by a technical "configuration problem" that needed fixing [38213]. 2. The NYSE experienced connectivity problems that may have prevented some customers from getting acknowledgments on orders submitted in some 220 stocks [38171]. 3. The NYSE had been experiencing technical issues even before the market opened, with some internet ports not connecting or unexpectedly disconnecting [38171]. 4. The NYSE had to manually cancel around 700,000 trades that were in the system when it was shut down [37661]. |
| Impacts | 1. The New York Stock Exchange experienced a technical "configuration problem" that led to an abrupt shutdown, causing trading to be suspended for more than three and a half hours, resulting in a decline in stocks by about 1.5% to 1.7% [38213, 38171]. 2. The outage raised concerns about the automation of Wall Street, highlighting the dependency on machines and the need for recovery systems in case of a true disaster [38213]. 3. Traders in China had a worse day as their stock market continued to decline, with some making quips about the NYSE trading halt [38213]. 4. The outage at the NYSE affected the trading of NYSE-listed stocks, but trading continued on other exchanges and private trading venues, minimizing the impact on investors [38171]. 5. The disruption rattled investors who were already on edge due to the Greek debt crisis and the Chinese stock market meltdown, leading to a 1.7% decline in the benchmark S&P 500 index [38171]. 6. The NYSE outage highlighted the fragility of automated systems in financial markets and raised questions about the robustness of technology at exchanges, especially after a series of major glitches in recent years [37661]. 7. The NYSE outage also affected other services on the same day, such as United Airlines grounding flights and the Wall Street Journal website experiencing technical problems, showcasing the broader impact of technology failures [37661]. |
| Preventions | 1. Regular testing and quality assurance of software updates to ensure they do not introduce bugs or errors [38213, 38171]. 2. Robust disaster recovery and contingency plans in place to quickly address technical issues and resume operations [38213]. 3. Enhanced monitoring and alert systems to detect and address technical problems before they escalate [38171]. 4. Investment in better technology and systems to prevent failures in automated exchanges [38171]. 5. Improved communication and transparency during incidents to keep stakeholders informed and mitigate concerns [37661]. |
| Fixes | 1. Investigating the technical "configuration problem" that caused the outage and making necessary changes to protocols [38213, 38213]. 2. Ensuring that software updates are thoroughly tested to prevent errors before implementation [38171]. 3. Implementing robust systems and technology to prevent future glitches and malfunctions [37661]. 4. Enhancing communication and response protocols in case of software failures to minimize disruptions [38213]. 5. Investing in new systems and technology expertise to improve trading platforms [37661]. | References | 1. NYSE president Thomas Farley [38213, 38171] 2. Homeland Security secretary Jeh Johnson [38213] 3. Wall Street Journal [38213] 4. Traders in China [38213] 5. Fortune Magazine [38213] 6. Eric Scott Hunsader, expert in Wall Street trading systems [38213] 7. The New York Times [38213] 8. Ted Weisberg, trader with Seaport Securities [37661] 9. Ryan Larson, head of stock trading at RBC Global Asset Management [37661] 10. Peter P. Costa, trader on the floor with Empire Executions [37661] 11. Mary Jo White, Chairwoman of the Securities and Exchange Commission [37661] 12. President Obama and White House officials [37661] 13. ICE spokeswoman, Sara Rich [37661] |
| Category | Option | Rationale |
|---|---|---|
| Recurring | one_organization, multiple_organization | (a) The software failure incident having happened again at one_organization: - The New York Stock Exchange (NYSE) experienced a software failure incident that led to a shutdown for several hours on July 8, 2015 [38213, 38171, 37661]. - The NYSE had faced technical issues in the past, such as a problem at NYSE's Arca exchange in March that caused some exchange-traded funds to be temporarily unavailable for trading [38171]. - The NYSE had been experiencing connectivity problems before the market opened on the day of the incident, affecting orders in some stocks [38171]. - The NYSE had to manually cancel around 700,000 trades that were in the system when it was shut down due to a software bug that could not be fixed without shutting down the system [37661]. (b) The software failure incident having happened again at multiple_organization: - The article mentions that the NYSE outage occurred on the same day as computer problems that grounded United Airlines flights and the homepage of The Wall Street Journal went down, indicating a broader issue with technology on that day [38171]. - The incident at the NYSE raised questions about the robustness of technology at exchanges in general, given a history of major glitches in recent years across various financial markets [38171]. - The Nasdaq stock market had experienced a three-hour outage in 2013 due to a software bug, and Knight Capital Group had faced software issues leading to significant losses in 2012, highlighting the vulnerability of financial markets to technical failures [37661]. |
| Phase (Design/Operation) | design | (a) The software failure incident at the New York Stock Exchange (NYSE) was attributed to a "configuration problem in our system that needed fixing" [38213]. NYSE president Tom Farley mentioned that the issue arose from a system update, but it was premature to confirm this [38213]. The exchange had been experiencing technical issues even before the market opened, with connectivity problems affecting orders in some stocks [38171]. The NYSE had to manually cancel around 700,000 trades that were in the system when it was shut down due to a bug in the software that required a system reboot [37661]. (b) The software failure incident at NYSE led to a halt in trading for several hours due to an internal technical issue, not a cyberattack [38171]. The exchange had to suspend trading because of an error with a systems upgrade that was rolled out before the market opened [38213]. The technical problem was identified and addressed, leading to the exchange reopening after resolving the issue [38213]. The failure raised questions about the robustness of technology at exchanges and the need for regulators to make markets more resistant to computer failings [37661]. |
| Boundary (Internal/External) | within_system | (a) within_system: The software failure incident at the New York Stock Exchange was attributed to an internal technical issue and a configuration problem within the system itself. NYSE President Thomas Farley mentioned that the exchange had been experiencing technical issues even before the market opened, with connectivity problems affecting orders in some stocks [Article 38171]. Additionally, it was reported that the software had a bug that required the system to be shut down for a fix, leading to the shutdown of trading for several hours [Article 37661]. (b) outside_system: There was no indication that the software failure incident at the NYSE was linked to external factors such as a cyberattack. Homeland Security Secretary Jeh Johnson stated that there was no indication of malicious activity, and officials monitored the situation for any signs of an outside attack, ultimately finding nothing indicating other than an internal software error [Article 38171]. |
| Nature (Human/Non-human) | non-human_actions, human_actions | (a) The software failure incident occurring due to non-human actions: - The New York Stock Exchange experienced a technical outage due to an internal technical issue, not a cyberattack [Article 38171]. - The NYSE glitch was attributed to a "configuration issue" and an "internal software error" rather than external malicious activity [Article 37661]. - The outage was caused by a "configuration problem in our system that needed fixing" and was not suspected to be external [Article 38213]. (b) The software failure incident occurring due to human actions: - The NYSE had been experiencing technical issues before the market opened, including connectivity problems affecting orders in some stocks [Article 38171]. - The exchange had updated software before the trading day began, which contained a bug that led to the shutdown [Article 37661]. - NYSE president Thomas Farley mentioned the need to change protocols and highlighted the importance of choosing the least disruptive option for customers, indicating potential human decisions contributing to the incident [Article 38213]. |
| Dimension (Hardware/Software) | hardware, software | (a) The articles provide information about a software failure incident related to hardware issues. The incident at the New York Stock Exchange (NYSE) on July 8, 2015, was attributed to an internal technical issue and not a cyberattack, indicating a hardware-related problem [Article 38171]. Traders on the floor mentioned that the exchange had been updating software before the trading day began, and the software contained a bug that required shutting down the system, pointing to a hardware-related issue [Article 37661]. (b) The articles also mention a software failure incident related to software issues. The NYSE president mentioned that the outage was caused by a "configuration problem in our system that needed fixing" [Article 38213]. Additionally, experts suggested that a faulty system upgrade brought trading on the exchange to a halt, indicating a software-related issue [Article 38171]. |
| Objective (Malicious/Non-malicious) | non-malicious | From the provided articles: (a) Malicious: - There was no indication that the outage at the New York Stock Exchange (NYSE) was linked to "nefarious" actors or a cyberattack, as confirmed by Homeland Security secretary Jeh Johnson and law enforcement officials [38213]. - The U.S. Department of Homeland Security stated that there were no signs that the problems at NYSE were due to malicious activity [38171]. (b) Non-malicious: - The outage at NYSE was attributed to an internal technical issue and not the result of a cyberattack [38171]. - NYSE president Thomas Farley mentioned that the outage was caused by a technical "configuration problem" that needed fixing, and there was no evidence to suspect external involvement [38213]. - The NYSE glitch was described as an internal software error, and federal regulators and law enforcement agencies monitored the situation for any sign of an outside attack but found nothing indicating other than an internal software error [37661]. |
| Intent (Poor/Accidental Decisions) | poor_decisions | (a) poor_decisions: The software failure incident at the New York Stock Exchange (NYSE) was attributed to a "configuration issue" that arose from an update to the system before the trading day began [Article 37661]. NYSE President Thomas Farley mentioned that the software had a bug that required shutting down the system, leading to the halt in trading [Article 37661]. The exchange's employees manually canceled around 700,000 trades that were in the system when it was shut down, indicating a need for better testing and quality control procedures [Article 37661]. (b) accidental_decisions: The outage at NYSE was described as an internal technical issue and not the result of a cyberattack [Article 38171]. The U.S. Department of Homeland Security stated that there were no signs of malicious activity related to the NYSE outage [Article 38171]. Additionally, the exchange's glitch was mentioned to be an error with a systems upgrade that was rolled out before the market opened on Wednesday [Article 38171]. |
| Capability (Incompetence/Accidental) | development_incompetence | (a) The software failure incident occurring due to development incompetence: - The New York Stock Exchange experienced a software failure incident that led to a shutdown for several hours. The incident was attributed to an internal technical issue and not a cyberattack [Article 38171]. - The exchange had been experiencing technical issues even before the market opened, with connectivity problems affecting orders in some stocks [Article 38171]. - The software failure incident at the NYSE was likely caused by a faulty system upgrade that brought trading to a halt [Article 38213]. - The NYSE had to manually cancel around 700,000 trades that were in the system when it was shut down, indicating a significant impact of the software failure [Article 37661]. (b) The software failure incident occurring accidentally: - The NYSE president mentioned that the outage was caused by a "configuration problem in our system that needed fixing" and that it was not external [Article 38213]. - The exchange had been updating software before the trading day began, and the software contained a bug that could not be fixed without shutting down the system, leading to the accidental failure [Article 37661]. - Federal regulators and law enforcement agencies monitored the situation for any sign of an outside attack but ultimately saw nothing indicating anything other than an internal software error, suggesting the incident was accidental [Article 37661]. |
| Duration | temporary | From the articles, it is evident that the software failure incident at the New York Stock Exchange was temporary rather than permanent. The outage lasted for several hours, with trading being suspended from around 11:30 a.m. EDT and resuming at 3:10 p.m. EDT [Article 38171]. The temporary nature of the failure is further supported by the fact that the exchange was able to manually cancel around 700,000 trades that were in the system when it was shut down and then reboot the system, leading to the resumption of trading within a relatively short period [Article 37661]. |
| Behaviour | crash, omission, timing, value, other | (a) crash: The software failure incident at the New York Stock Exchange (NYSE) can be categorized as a crash. The incident led to the abrupt shutdown of the NYSE at 11:32 am local time, and trading did not resume until about 3:10 pm [38213]. The system lost its state and was not performing its intended functions during this period, resulting in a significant outage [38171]. (b) omission: The software failure incident can also be categorized as an omission. Before the market opened, the NYSE was experiencing connectivity problems that prevented some customers from receiving acknowledgments on orders submitted for certain stocks [38171]. This omission to perform its intended functions affected the trading process. (c) timing: The software failure incident can be categorized as a timing issue as well. The NYSE had been experiencing technical issues even before the market opened, with connectivity problems affecting orders for some stocks [38171]. This issue occurred at a critical time, impacting the trading process. (d) value: The software failure incident can be categorized as a value issue. The NYSE had to manually cancel around 700,000 trades that were in the system when it was shut down due to a software bug that could not be fixed without shutting down the system [37661]. This incorrect performance of the system led to the need to cancel trades to rectify the situation. (e) byzantine: The software failure incident does not align with a byzantine behavior as there is no indication of inconsistent responses or interactions in the articles provided. (f) other: The software failure incident can be categorized as a "configuration problem" that led to the outage at the NYSE [38213]. This configuration issue caused the system to fail and required fixing, indicating a specific type of failure related to the system's setup or settings. |
| Layer | Option | Rationale |
|---|---|---|
| Perception | None | None |
| Communication | None | None |
| Application | None | None |
| Category | Option | Rationale |
|---|---|---|
| Consequence | property, delay, non-human, theoretical_consequence | (a) death: People lost their lives due to the software failure - No mention of any deaths due to the software failure incident in the articles. [38213, 38171, 37661] (b) harm: People were physically harmed due to the software failure - No mention of physical harm to individuals due to the software failure incident. [38213, 38171, 37661] (c) basic: People's access to food or shelter was impacted because of the software failure - No mention of people's access to food or shelter being impacted by the software failure incident. [38213, 38171, 37661] (d) property: People's material goods, money, or data was impacted due to the software failure - The software failure incident impacted trading on the New York Stock Exchange, causing disruptions in the stock market and affecting investors. [38213, 38171, 37661] (e) delay: People had to postpone an activity due to the software failure - Traders had to wait for the NYSE to resume trading after the software failure incident, causing delays in trading activities. [38213, 38171, 37661] (f) non-human: Non-human entities were impacted due to the software failure - The software failure incident affected the operations of the New York Stock Exchange, leading to a halt in trading and technical issues. [38213, 38171, 37661] (g) no_consequence: There were no real observed consequences of the software failure - The software failure incident had significant consequences on trading activities and market operations, as detailed in the articles. [38213, 38171, 37661] (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur - The articles discuss potential consequences of the software failure, such as the impact on closing auctions, market stability, and the need for recovery systems in case of a disaster. [38213, 38171, 37661] (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? - No other specific consequences beyond those mentioned in the options were described in the articles. [38213, 38171, 37661] |
| Domain | finance | (a) The failed system was intended to support the finance industry, specifically the stock market. The New York Stock Exchange (NYSE) experienced a technical outage that led to the suspension of trading for several hours, causing disruptions in the financial market [38213, 38171, 37661]. The outage affected trading of stocks and impacted investors and traders who rely on the NYSE for transactions and end-of-day pricing [38171]. The incident raised concerns about the automation of Wall Street and the dependency on technology for trading activities [38213, 38171]. The NYSE president mentioned that the outage was due to an internal technical issue and not a cyberattack [38171]. The Department of Homeland Security also stated that there were no signs of malicious activity related to the NYSE problem [38171]. The NYSE's glitch was part of a series of technical issues experienced by major exchanges in recent years, highlighting the importance of robust technology in financial markets [38171]. |
Article ID: 38213
Article ID: 38171
Article ID: 37661