Incident: HSBC Online Banking Outage Impacting Millions of Customers.

Published Date: 2016-01-05

Postmortem Analysis
Timeline 1. The software failure incident at HSBC happened on Monday, as mentioned in the article [39580]. 2. The article was published on 2016-01-05. 3. Therefore, the software failure incident at HSBC occurred on Monday, January 4, 2016.
System 1. HSBC online banking system 2. HSBC mobile banking system 3. Personal Banking mobile app
Responsible Organization 1. HSBC's internet banking systems [39580]
Impacted Organization 1. HSBC customers [39580]
Software Causes 1. Complex technical issue with HSBC's internet banking systems [39580] 2. IT glitch in HSBC's systems [39580]
Non-software Causes 1. High demand during a traditionally busy time for banks due to customers checking their bank balances after the Christmas and New Year celebrations [39580]. 2. Complex technical issue with HSBC's internet banking systems [39580].
Impacts 1. Millions of HSBC customers were unable to log on to their personal banking website, causing inconvenience and frustration [39580]. 2. Up to 17 million personal and business customers were locked out of their accounts for up to nine hours, disrupting their ability to carry out online transactions [39580]. 3. Customers experienced delays in accessing online services during a traditionally busy time for banks, affecting their ability to check their bank balances after the holiday season [39580]. 4. HSBC had to waive any fees incurred by customers as a result of the outage, indicating a financial impact on both the bank and its customers [39580]. 5. The incident led to increased demand on call centers and branches, requiring mobilization of resources to cope with customer inquiries and complaints [39580].
Preventions 1. Implementing robust testing procedures before deploying any updates or changes to the online banking system could have prevented the software failure incident [39580]. 2. Conducting regular maintenance and monitoring of the online banking system to identify and address any potential issues before they escalate into major disruptions [39580]. 3. Enhancing the capacity and scalability of the online banking infrastructure to handle peak usage periods, such as the first day back at work after holidays, to prevent system overloads and outages [39580].
Fixes 1. Implementing a comprehensive solution to address the complex technical issue with the internet banking systems at HSBC, as mentioned by John Hackett, the UK chief operating officer [39580]. 2. Conducting thorough testing, diagnostics, and trial runs by the IT team at HSBC to find a solution to the software failure incident [39580]. 3. Providing regular updates to customers about the progress in resolving the online banking issues at HSBC to maintain transparency and keep customers informed [39580]. 4. Mobilizing all available resources, including call centers and branches, to cope with the increased customer demand and provide support during the software failure incident at HSBC [39580].
References 1. Twitter users 2. HSBC spokesperson 3. John Hackett, HSBC’s UK chief operating officer 4. HSBC's official Twitter account 5. NatWest and Royal Bank of Scotland customers [39580]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: - HSBC experienced a major IT glitch in August 2015 when 275,000 bank payments failed to go through just before the bank holiday weekend. This incident is similar to the recent online banking disruption where millions of customers were locked out of their accounts for up to nine hours [39580]. (b) The software failure incident having happened again at multiple_organization: - Last week, NatWest and Royal Bank of Scotland customers also faced a banking glitch affecting debit card transactions in shops, where cards were declined at tills and pins were blocked. This incident shows a similar issue occurring at multiple organizations [39580].
Phase (Design/Operation) design, operation (a) The software failure incident at HSBC was primarily due to issues related to system development and updates. The article mentions a "major IT glitch in August 2015" where 275,000 bank payments failed to go through just before a bank holiday weekend. Additionally, John Hackett, HSBC’s UK chief operating officer, stated that there was a "complex technical issue with our internet banking systems" that the IT team has been working on to find a solution [39580]. These instances point towards failures introduced during the development and maintenance phases of the system. (b) The software failure incident at HSBC also had elements related to operation and customer misuse. Customers were unable to log in to their accounts, and there were complaints about the online services being down, impacting users' ability to carry out transactions and check their balances. The article mentions that customers were venting their anger due to the banking glitch affecting debit card usage in shops, with cards being declined at tills and pins being blocked [39580]. These issues highlight the operational aspect of the software failure incident.
Boundary (Internal/External) within_system (a) within_system: The software failure incident at HSBC was due to a complex technical issue with their internet banking systems, as mentioned by John Hackett, HSBC’s UK chief operating officer. The IT team at HSBC has been working non-stop to find a solution, involving tests, diagnostics, and trial runs. It was clarified that the issue was not a cyber-attack or any other malicious act, indicating that the problem originated within the system [39580]. (b) outside_system: There is no specific mention in the articles about the software failure incident at HSBC being caused by contributing factors originating from outside the system.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident at HSBC was attributed to a complex technical issue with their internet banking systems, as mentioned by John Hackett, HSBC’s UK chief operating officer. He clarified that it was not a cyber-attack or any other malicious act, indicating that the failure was due to contributing factors introduced without human participation [39580]. (b) On the other hand, the article also mentions that HSBC had experienced a major IT glitch in August 2015 when 275,000 bank payments failed to go through just before the bank holiday weekend. This incident could be attributed to human actions, such as potential errors in system updates, maintenance, or configuration changes made by IT personnel [39580].
Dimension (Hardware/Software) software (a) The software failure incident occurring due to hardware: - The article does not mention any specific hardware-related issues contributing to the software failure incident at HSBC [39580]. (b) The software failure incident occurring due to software: - The software failure incident at HSBC was attributed to a complex technical issue with their internet banking systems, as stated by John Hackett, HSBC’s UK chief operating officer [39580].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident at HSBC was classified as non-malicious. The article mentions that the issue was due to a complex technical issue with the internet banking systems, and HSBC's UK chief operating officer confirmed that it was not a cyber-attack or any other malicious act [39580]. (b) The incident at NatWest and Royal Bank of Scotland, where customers faced issues with debit cards in shops, could potentially be classified as non-malicious as well, as there is no indication in the provided article that the glitch was caused by malicious intent.
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident at HSBC seems to be more related to poor_decisions rather than accidental_decisions. The incident was described as a major IT glitch that caused disruptions to online banking services for millions of customers. HSBC's UK chief operating officer, John Hackett, mentioned that there was a complex technical issue with their internet banking systems that the IT team had been working non-stop to resolve. Additionally, HSBC had faced a similar IT glitch in August 2015 where 275,000 bank payments failed to go through, indicating a recurring issue that may have been exacerbated by poor decisions in the management of their IT systems [39580].
Capability (Incompetence/Accidental) accidental (a) The software failure incident at HSBC was not attributed to development incompetence. The article mentions that the issue was due to a complex technical problem with the internet banking systems, and the IT team was working non-stop to find a solution. HSBC's UK chief operating officer mentioned that it was not a cyber-attack or any other malicious act [39580]. (b) The software failure incident at HSBC was described as a major IT glitch, indicating that it was an accidental failure rather than a deliberate act of incompetence. The article mentions that the bank was experiencing further issues with online and mobile banking, and the IT team was working to find a solution. The chief operating officer mentioned that the problem was not caused by a cyber-attack or any other malicious act [39580].
Duration temporary (a) The software failure incident reported in the articles was temporary. The incident caused online banking services for millions of HSBC customers to be disrupted for several hours on Monday, with customers being locked out of their accounts for up to nine hours [39580]. Additionally, on Tuesday, customers were still experiencing issues with the bank's internet services, indicating a temporary disruption rather than a permanent failure. The bank acknowledged the ongoing problems with online and mobile banking but assured customers that efforts were being made to restore all services and that regular updates would be provided [39580].
Behaviour crash, omission, timing, other (a) crash: The software failure incident described in the articles can be categorized as a crash. The HSBC online banking system experienced a significant disruption where customers were unable to log in for several hours, indicating a failure of the system to perform its intended functions [39580]. (b) omission: The incident also involved omission as customers were locked out of their accounts and unable to access online banking services, leading to the system omitting to perform its intended functions [39580]. (c) timing: The timing of the software failure incident is also relevant as it occurred during a busy period when customers were returning to work after the holidays and needed to check their bank balances. The system's failure to function correctly at this critical time highlights a timing-related issue [39580]. (d) value: There is no specific mention of the system performing its intended functions incorrectly in the articles. (e) byzantine: The incident does not exhibit characteristics of a byzantine failure where the system behaves erroneously with inconsistent responses and interactions. (f) other: The other behavior observed in this software failure incident is the system experiencing a complex technical issue with its internet banking systems, requiring extensive testing and diagnostics to find a solution. This behavior falls under the "other" category as it involves a specific technical problem not covered by the other options [39580].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay, non-human, theoretical_consequence (a) death: There is no mention of any deaths resulting from the software failure incident in the provided articles [39580]. (b) harm: There is no mention of physical harm to individuals resulting from the software failure incident in the provided articles [39580]. (c) basic: There is no mention of people's access to food or shelter being impacted due to the software failure incident in the provided articles [39580]. (d) property: The software failure incident impacted people's access to their accounts and online banking services, potentially affecting their financial transactions and access to funds [39580]. (e) delay: Customers experienced delays in accessing online banking services and conducting transactions due to the software failure incident [39580]. (f) non-human: The software failure incident affected the functionality of HSBC's online banking systems and services, impacting the bank's operations and customer service [39580]. (g) no_consequence: The software failure incident led to real consequences for customers, such as being unable to access online banking services and facing delays in transactions [39580]. (h) theoretical_consequence: There were potential consequences discussed, such as customers incurring fees as a result of the outage, but HSBC stated that any fees incurred would be waived [39580]. (i) other: There is no mention of other specific consequences resulting from the software failure incident in the provided articles [39580].
Domain finance (a) The failed system was related to the finance industry as it affected HSBC's online banking services, causing disruptions for millions of customers [39580]. Additionally, the incident mentions that customers were unable to access their accounts and make payments online, highlighting the financial nature of the system failure.

Sources

Back to List