Incident: Nasdaq Stock Market Trading Halted Due to Software Glitch

Published Date: 2013-08-23

Postmortem Analysis
Timeline 1. The software failure incident happened on August 22, 2013. [20950] 2. The software failure incident happened on August 22, 2013. [20719]
System 1. The software failure incident at Nasdaq was attributed to a glitch in the software that publishes the prices of stocks listed on the exchange [20719, 20950].
Responsible Organization 1. Nasdaq - The software failure incident was caused by a glitch in software that publishes the prices of stocks listed on the Nasdaq exchange [20719, 20950].
Impacted Organization 1. Nasdaq exchange [20950, 20719] 2. Traders and investors involved in trading Nasdaq-listed stocks [20950, 20719] 3. Other exchanges that depend on Nasdaq's pricing software for accuracy [20719]
Software Causes 1. The failure incident at Nasdaq was caused by a glitch in the software that publishes the prices of stocks listed on the exchange [20719]. 2. The issues at Nasdaq were attributed to technical issues at a central clearinghouse known as the "UTP SIP," which is the security information process where 13 public exchanges send bid and offer prices on Nasdaq-listed stocks [20950].
Non-software Causes 1. The failure incident at Nasdaq was attributed to technical issues at a central clearinghouse known as the "UTP SIP" [20950]. 2. The glitch in the Nasdaq stock market resulted from a glitch in software that publishes the prices of stocks listed on the exchange [20719].
Impacts 1. The software failure incident led to the halt of trading in more than 2,700 Nasdaq-listed stocks for several hours, affecting major companies like Apple, Google, Microsoft, Intel, and Facebook [20950]. 2. The outage resulted in confusion and calls for greater regulation in the stock market, with Nasdaq shutting down for over three hours, impacting about 20% of the S&P 500 companies listed on Nasdaq [20719]. 3. The glitch in the software that publishes stock prices on Nasdaq caused a lock-up of roughly $5.7 trillion in shares across various exchanges that depend on Nasdaq's pricing software for accuracy [20719]. 4. Investors were advised to cancel their own trades before the system went back up, and the Securities and Exchange Commission was monitoring the situation closely [20719]. 5. The failure refocused attention on a series of prominent failures in market technology, including other recent glitches like the mistaken options trade by Goldman Sachs earlier in the week [20719].
Preventions 1. Implementing robust testing procedures: Conducting thorough testing of software systems, including stress testing and scenario testing, could have potentially identified and addressed the glitch in the software that caused the outage [20719]. 2. Regular software maintenance and updates: Ensuring that the software systems are regularly maintained and updated with the latest patches and fixes could have helped prevent the glitch that led to the trading outage [20719]. 3. Enhanced regulatory oversight: Strengthening regulatory efforts, such as implementing Regulation SCI, which would require major exchanges to submit to audits of their technological systems, could have helped in monitoring and preventing such software failures in the future [20719].
Fixes 1. Implementing stricter regulations and oversight on stock exchanges' technological systems to ensure they are robust and reliable [Article 20719]. 2. Requiring major exchanges to submit to audits of their backup plans and technological systems, potentially through Regulation SCI [Article 20719]. 3. Enhancing monitoring of exchanges by regulators to keep up with the speed of current technology and prevent future failures [Article 20719].
References 1. Nasdaq (NDAQ) - The articles mention Nasdaq as the entity that experienced the software failure incident [20950, 20719]. 2. Securities and Exchange Commission (SEC) - The articles refer to the SEC as the regulatory body overseeing the stock exchanges and monitoring the situation [20719]. 3. Market experts and consultants - The articles quote market experts and consultants like Chris Nagy and Dennis Dick who provide insights into the software failure incident [20950, 20719]. 4. Traders and market participants - The articles include statements from traders and market participants who were directly impacted by the software failure incident [20950, 20719]. 5. Other exchanges and financial institutions - The articles mention that the software failure incident at Nasdaq affected other exchanges and financial institutions that rely on Nasdaq's pricing software [20719].

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: - Nasdaq experienced a glitch in its software that resulted in a shutdown of trading for over 2,000 major US stocks and options [Article 20719]. - Nasdaq had previously faced trading issues at its exchange that marred the initial public offering of Facebook [Article 20950]. (b) The software failure incident having happened again at multiple_organization: - Goldman Sachs also encountered a massive trading glitch due to an internal computer system problem that flooded the options markets with erroneous orders [Article 20950]. - Knight Capital experienced a high-profile trading glitch that resulted in a loss of $440 million after a software snafu affected trading activity in nearly 150 NYSE-listed stocks [Article 20950]. - Everbright Securities in China faced problems that caused a sudden 6% spike in the Shanghai Composite Index due to trading glitches [Article 20950].
Phase (Design/Operation) design, operation (a) The software failure incident related to the design phase: - The incident was attributed to a glitch in software that publishes the prices of stocks listed on the exchange, affecting over 2,000 major US stocks and options [Article 20719]. - Nasdaq experienced a failure resulting from technical issues at a central clearinghouse known as the "UTP SIP," where 13 public exchanges send bid and offer prices on Nasdaq-listed stocks [Article 20950]. (b) The software failure incident related to the operation phase: - The outage affected a host of other exchanges as they depend on Nasdaq's pricing software for accuracy, locking up roughly $5.7 trillion in shares [Article 20719]. - Traders and market structure consultants noticed the problems midday when they stopped seeing quotes in certain Nasdaq-listed stocks, leading to the halt in trading [Article 20950].
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident related to the Nasdaq stock market shutdown was attributed to a glitch in the software that publishes the prices of stocks listed on the exchange [20719]. Nasdaq mentioned that the failure resulted from a glitch in their software [20719]. The issues were specifically related to the software used for pricing stocks listed on the exchange [20719]. (b) outside_system: The software failure incident was also influenced by external factors such as the impact on other exchanges that depend on Nasdaq's pricing software for accuracy [20719]. The outage affected a host of other exchanges, ranging from NYSE Euronext to systems that serve primarily professional investors who trade in large blocks of thousands of shares at a time [20719]. The halt locked up roughly $5.7 trillion in shares [20719].
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident occurring due to non-human actions: - The software failure incident at Nasdaq was attributed to a glitch in the software that publishes the prices of stocks listed on the exchange [Article 20719]. - Nasdaq attributed the problems to technical issues at a central clearinghouse known as the "UTP SIP," which is the security information process where 13 public exchanges send bid and offer prices on Nasdaq-listed stocks [Article 20950]. (b) The software failure incident occurring due to human actions: - An internal computer system problem caused Goldman Sachs to flood the options markets with erroneous orders, leading to a massive trading glitch [Article 20950]. - Earlier in the week, Goldman Sachs made a mistaken options trade that rippled across the markets and affected a number of financial institutions [Article 20719].
Dimension (Hardware/Software) software (a) The software failure incident occurring due to hardware: - The incident reported in the articles is primarily attributed to a glitch in software that publishes the prices of stocks listed on the Nasdaq exchange [20719]. - Nasdaq experienced an unprecedented shutdown of trading due to a glitch in software that caused the outage [20719]. - The failure at Nasdaq resulted from a glitch in software, specifically in the software that publishes the prices of stocks listed on the exchange [20719]. (b) The software failure incident occurring due to software: - The software failure incident is directly linked to a glitch in software that publishes the prices of stocks listed on the Nasdaq exchange [20719]. - Nasdaq attributed the problems to technical issues at a central clearinghouse known as the "UTP SIP," which is the security information process where 13 public exchanges send bid and offer prices on Nasdaq-listed stocks [20950]. - The failure at Nasdaq resulted from a glitch in software, specifically in the software that publishes the prices of stocks listed on the exchange [20719].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident reported in the articles appears to be non-malicious. The incidents were attributed to technical glitches in the software that publishes the prices of stocks listed on the Nasdaq exchange [20719]. Nasdaq experienced a shutdown in trading due to a glitch in the software, causing confusion and affecting the trading of major US stocks and options [20719]. The failure was not attributed to any malicious intent but rather to a technical issue within the software system. Additionally, the articles mention other recent trading glitches in the financial markets, such as the trading issues at Goldman Sachs and the flash crash in 2010, which were also non-malicious in nature [20950]. These incidents highlight the complexity and interconnectedness of modern trading systems, where even small technical hiccups can have significant impacts on the market. Therefore, based on the information provided in the articles, the software failure incidents discussed were non-malicious in nature, stemming from technical issues rather than intentional harm to the system.
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident related to the Nasdaq stock market shutdown was primarily due to poor decisions. The incident was attributed to a glitch in the software that publishes the prices of stocks listed on the exchange [20719]. Nasdaq's failure to cancel pending stock trades automatically and instead advising investors to cancel their own trades before the system goes back up could be seen as a poor decision that added to the confusion and impact of the incident [20719]. Additionally, the incident raised calls for greater regulation, indicating that there were shortcomings in the decision-making processes related to market technology and regulatory oversight [20719].
Capability (Incompetence/Accidental) development_incompetence (a) The software failure incident occurring due to development incompetence: - The incident at Nasdaq was attributed to technical issues at a central clearinghouse known as the "UTP SIP," where bid and offer prices on Nasdaq-listed stocks are sent [20950]. - The failure resulted from a glitch in software that publishes the prices of stocks listed on the exchange [20719]. (b) The software failure incident occurring accidentally: - Nasdaq hasn't revealed what caused the issues at the central clearinghouse "UTP SIP" [20950]. - Nasdaq stated that the failure resulted from a glitch in software that publishes the prices of stocks listed on the exchange [20719].
Duration temporary (a) The software failure incident in the articles was temporary. Nasdaq experienced a glitch in its software that resulted in an unprecedented shutdown of trading for over 2,000 major US stocks and options for more than two hours [20719]. The outage lasted for over three hours as Nasdaq investigated the issue, and trading resumed at 3:25 pm ET [20719]. (b) The software failure incident was not permanent as it was resolved within a few hours, indicating that the contributing factors introduced by certain circumstances were addressed, leading to the restoration of trading operations.
Behaviour crash, omission, value, other (a) crash: The software failure incident mentioned in the articles can be categorized as a crash. The incident led to a significant shutdown of trading in over 2,000 major US stocks and options on the Nasdaq stock market for several hours, causing confusion and calls for greater regulation [20719]. Nasdaq attributed the problems to technical issues in the software that publishes the prices of stocks listed on the exchange, resulting in a halt in trading until the issue was resolved [20719]. (b) omission: The software failure incident can also be categorized as an omission. Due to the glitch in the software that publishes stock prices on Nasdaq, the system omitted to perform its intended functions, leading to the shutdown of trading for more than two hours and affecting other exchanges that depend on Nasdaq's pricing software for accuracy [20719]. (c) timing: The software failure incident does not align with a timing failure as the system did not perform its intended functions too late or too early. The issue was more about the system not functioning as expected, leading to a shutdown in trading [unknown]. (d) value: The software failure incident can be associated with a value failure as the system performed its intended functions incorrectly, resulting in the glitch that caused the trading shutdown on Nasdaq and affected other exchanges as well [20719]. (e) byzantine: The software failure incident does not exhibit characteristics of a byzantine failure where the system behaves erroneously with inconsistent responses and interactions. The issue was more focused on a glitch in the software that disrupted trading activities [unknown]. (f) other: The software failure incident can be further described as a failure caused by technical issues at a central clearinghouse known as the "UTP SIP," affecting the bid and offer prices on Nasdaq-listed stocks. This specific technical issue led to the crash in trading and the subsequent shutdown until the problem was resolved [20950].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay, non-human, other (a) death: People lost their lives due to the software failure - No mention of any deaths due to the software failure incident in the articles [20950, 20719]. (b) harm: People were physically harmed due to the software failure - No mention of physical harm to individuals due to the software failure incident in the articles [20950, 20719]. (c) basic: People's access to food or shelter was impacted because of the software failure - No mention of people's access to food or shelter being impacted due to the software failure incident in the articles [20950, 20719]. (d) property: People's material goods, money, or data was impacted due to the software failure - The software failure incident impacted trading in major stocks, causing confusion and financial implications for investors and traders [20950, 20719]. (e) delay: People had to postpone an activity due to the software failure - Trading in over 2,000 major US stocks and options was halted for more than two hours due to the software failure incident, causing disruption and confusion in the market [20719]. (f) non-human: Non-human entities were impacted due to the software failure - The software failure incident affected the trading of stocks and options on major exchanges, leading to a halt in trading and financial implications [20950, 20719]. (g) no_consequence: There were no real observed consequences of the software failure - The software failure incident had significant consequences on trading activities, market stability, and investor confidence [20950, 20719]. (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur - The articles do not mention potential consequences that were discussed but did not occur due to the software failure incident [20950, 20719]. (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? - The software failure incident resulted in a halt in trading, confusion among investors, financial implications, and calls for greater regulation in the market [20950, 20719].
Domain finance (a) The failed system was intended to support the finance industry. The software failure incident was related to the stock market, trading glitches, and the outage at Nasdaq, which is a major stock exchange where companies like Apple and Facebook are listed [20950, 20719]. (h) The failed system was also related to the finance industry as it impacted trading activities, caused confusion around shares of major companies, and resulted in calls for greater regulation in the stock market [20950, 20719]. (m) The software failure incident was not related to any other industry mentioned in the options provided.

Sources

Back to List