Incident: Trading Halted on Tokyo Stock Exchange Due to Hardware Failure

Published Date: 2020-09-30

Postmortem Analysis
Timeline 1. The software failure incident at the Tokyo Stock Exchange happened on October 1, 2020 [106973, 105591].
System [106973, 105591] The hardware failure that caused the software failure incident at the Tokyo Stock Exchange was related to the "Arrowhead" trading system developed by Fujitsu Ltd.
Responsible Organization 1. The software failure incident at the Tokyo Stock Exchange was caused by a hardware problem at its "Arrowhead" trading system, developed by Fujitsu Ltd [106973, 105591].
Impacted Organization 1. Investors and market participants were impacted by the software failure incident at the Tokyo Stock Exchange [106973, 105591].
Software Causes 1. The software failure incident at the Tokyo Stock Exchange was caused by a hardware problem at its "Arrowhead" trading system, leading to a subsequent failure to switch to a back-up system [106973, 105591].
Non-software Causes 1. Hardware failure at the Tokyo Stock Exchange's "Arrowhead" trading system [106973, 105591] 2. Failure to switch to a back-up system [106973, 105591]
Impacts 1. The software failure incident at the Tokyo Stock Exchange caused a complete standstill in trading, leading to the worst-ever outage for the world's third-largest equity market [106973, 105591]. 2. Frustrated investors were unable to buy shares in Tokyo following an overnight rise on Wall Street, impacting their ability to make timely investment decisions [106973, 105591]. 3. The incident tested the exchange's credibility and could potentially dent Tokyo's efforts to attract banks and fund managers from Hong Kong [106973, 105591]. 4. The glitch was attributed to a hardware problem at the "Arrowhead" trading system, which failed to switch to a backup, resulting in the first full-day suspension since the exchange switched to all-electronic trading in 1999 [106973, 105591]. 5. The outage affected not only the Tokyo Stock Exchange but also smaller bourses in Nagoya, Fukuoka, and Sapporo that use the same trading system [106973, 105591]. 6. The incident highlighted concerns about the TSE's ability to handle large trading volumes, especially during critical events like the U.S. presidential election [106973, 105591].
Preventions 1. Implementing a more robust hardware system with better redundancy and failover mechanisms could have prevented the software failure incident [106973, 105591]. 2. Conducting regular and thorough testing of the trading system, including testing failover procedures, could have helped identify and address potential issues before they caused a complete standstill [106973, 105591]. 3. Enhancing monitoring and alerting systems to quickly detect hardware failures and automatically switch to backup systems could have minimized the impact of the incident [106973, 105591]. 4. Establishing a clear and well-documented incident response plan for hardware failures, including detailed steps for troubleshooting and recovery, could have facilitated a faster resolution of the issue [106973, 105591].
Fixes 1. Implementing a quick fix to ensure trust in the market and prevent future incidents [106973, 105591] 2. Investigating the hardware problem at the "Arrowhead" trading system and ensuring a proper switch to a backup system [106973, 105591] 3. Ensuring that the TSE system does not face similar technical troubles in the future [106973, 105591]
References 1. TSE Chief Executive Officer Koichiro Miyahara 2. Fujitsu Ltd 3. Tokyo Governor Yuriko Koike 4. William de Gale, manager of the UK-based BlueBox Global Technology Fund 5. Japan's top government spokesman, Katsunobu Kato 6. Hideyuki Ishiguro, a senior strategist at Daiwa Securities 7. Various reporters and journalists mentioned in the articles [106973, 105591]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: The Tokyo Stock Exchange (TSE) has experienced technical troubles in the past and was notorious for sluggish trading. However, there have been fewer glitches since a new system was adopted in 2010 [105591]. This suggests that the TSE has faced software failure incidents before but has taken steps to improve its systems. (b) The software failure incident having happened again at multiple_organization: The article mentions that the most recent serious outage at a major exchange was when Germany's electronic trading platform Xetra, managed by Deutsche Boerse, had blackouts in April and again in July. Additionally, a technical glitch caused the London Stock Exchange to have its longest outage in years in 2019 [105591]. This indicates that software failure incidents have occurred at other major exchanges as well.
Phase (Design/Operation) design (a) The software failure incident at the Tokyo Stock Exchange was attributed to a hardware problem in its "Arrowhead" trading system, developed by Fujitsu Ltd. The glitch was a result of a hardware failure and a subsequent failure to switch to a backup system. This indicates a failure in the design phase, where contributing factors introduced during system development led to the outage [106973, 105591]. (b) The articles do not provide specific information indicating that the software failure incident was due to factors introduced by the operation or misuse of the system.
Boundary (Internal/External) within_system (a) within_system: The software failure incident at the Tokyo Stock Exchange was attributed to a hardware problem within the TSE's "Arrowhead" trading system, which led to a failure to switch to a backup system [106973, 105591]. This indicates that the failure originated from within the system itself, specifically from a hardware issue within the trading system.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident at the Tokyo Stock Exchange was primarily attributed to a hardware failure in the "Arrowhead" trading system, which was a non-human action [106973, 105591]. The glitch was a result of a hardware problem and a subsequent failure to switch to a backup system, causing the first full-day suspension since the exchange switched to all-electronic trading in 1999. Fujitsu Ltd, the developer of the trading system, was investigating the issue, indicating that the failure was not directly caused by human actions. (b) While the software failure incident was primarily due to a hardware failure, there were implications of human actions in terms of responsibility and accountability. The TSE Chief Executive Officer expressed feeling responsible for the confusion caused to investors and market participants [106973, 105591]. Additionally, there were discussions about the need to ensure such incidents are never repeated, suggesting a human element in preventing future failures.
Dimension (Hardware/Software) hardware (a) The software failure incident occurred due to hardware issues. The Tokyo Stock Exchange experienced a complete standstill in trading due to a hardware failure in its "Arrowhead" trading system, which resulted in the first full-day suspension since the exchange switched to all-electronic trading in 1999 [106973, 105591]. (b) The software failure incident was not directly attributed to software issues but rather to a hardware problem in the trading system. Fujitsu Ltd, the company that developed the trading system, was investigating the problem, indicating that the root cause was related to hardware rather than software [106973, 105591].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident at the Tokyo Stock Exchange was non-malicious. The incident was attributed to a hardware problem at the "Arrowhead" trading system, which led to a failure to switch to a backup system, causing the first full-day suspension of trading since the exchange switched to all-electronic trading in 1999 [106973, 105591]. Fujitsu Ltd, the developer of the trading system, was investigating the problem, and there was no mention of any intentional malicious activity causing the failure. The focus was on resolving the technical issue and ensuring trust in the market [106973, 105591].
Intent (Poor/Accidental Decisions) unknown (a) The software failure incident at the Tokyo Stock Exchange was primarily attributed to a hardware failure in the "Arrowhead" trading system, which led to a subsequent failure to switch to a backup system [106973, 105591]. This indicates that the incident was not caused by poor decisions but rather by a technical issue with the hardware system.
Capability (Incompetence/Accidental) development_incompetence (a) The software failure incident at the Tokyo Stock Exchange was attributed to a hardware problem in its "Arrowhead" trading system, which led to a failure to switch to a backup system [106973, 105591]. This indicates a failure due to development incompetence, as the issue stemmed from a technical flaw in the system developed by Fujitsu Ltd. The incident highlighted the importance of professional competence in developing and maintaining critical trading systems to prevent such failures. (b) The articles do not provide specific information indicating that the software failure incident was accidental.
Duration temporary The software failure incident at the Tokyo Stock Exchange was temporary. The trading halt was caused by a hardware failure at the TSE's "Arrowhead" trading system, which led to the first full-day suspension of trading since the exchange switched to all-electronic trading in 1999 [106973, 105591]. The incident was not permanent as it was a result of specific contributing factors related to the hardware problem and the failure to switch to a backup system, rather than being a systemic issue affecting all circumstances.
Behaviour crash, omission, other (a) crash: The software failure incident in the Tokyo Stock Exchange was described as a crash where trading was brought to a complete standstill due to a hardware failure, causing the first full-day suspension since the exchange switched to all-electronic trading in 1999 [106973, 105591]. (b) omission: The software failure incident resulted in the omission of the system to perform its intended functions, as frustrated investors were left unable to buy shares in Tokyo following an overnight rise on Wall Street [106973, 105591]. (c) timing: The timing of the software failure incident was crucial as it occurred just when many market participants were hoping to buy back their stocks or increase their holdings after an overnight rise in U.S. markets, but were unable to do so due to the system failure [106973, 105591]. (d) value: There is no specific mention of the software failure incident being related to the system performing its intended functions incorrectly in the articles. (e) byzantine: The software failure incident did not exhibit behavior related to the system behaving erroneously with inconsistent responses and interactions in the articles. (f) other: The software failure incident also led to the suspension of trading at smaller bourses in Nagoya, Fukuoka, and Sapporo because they use the same TSE system, highlighting the widespread impact of the failure [105591].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property (d) property: People's material goods, money, or data was impacted due to the software failure The software failure incident at the Tokyo Stock Exchange caused a complete standstill in trading due to a hardware failure, leading to frustrated investors being unable to buy shares in Tokyo. This impacted investors' ability to trade and potentially affected their financial assets [106973, 105591].
Domain finance (a) The failed system was related to the finance industry as it impacted the Tokyo Stock Exchange, which is a major equity market with a daily turnover of shares traded on the TSE's main board averaging about 2.33 trillion yen ($22 billion) over the past year [106973, 105591]. The incident caused a complete standstill in trading due to a hardware failure in the TSE's "Arrowhead" trading system, leading to the first full-day suspension since the exchange switched to all-electronic trading in 1999 [106973, 105591]. The glitch affected investors and market participants, highlighting the importance of a quick fix to ensure trust in the market [106973, 105591]. The outage also coincided with the release of the Bank of Japan's tankan corporate survey, showing an improvement in business sentiment from a decade low [106973, 105591]. The incident raised concerns about the TSE's ability to handle a surge in trading volumes, especially around events like the U.S. presidential election [106973]. (h) The failed system was specifically designed for the finance industry, supporting stock trading activities on the Tokyo Stock Exchange [106973, 105591].

Sources

Back to List