Incident: HSBC IT Glitch Causes Payment Failures Before Bank Holiday

Published Date: 2015-08-28

Postmortem Analysis
Timeline 1. The software failure incident at HSBC, where thousands of Britons failed to receive their wages, happened on a Friday before the bank holiday weekend in August [38843, 38695]. 2. Article 38843 was published on 2015-08-28. 3. Therefore, the software failure incident occurred in August 2015.
System 1. Bacs system - The Bacs system, which processes electronic transactions, failed in the software failure incident at HSBC [38843, 38695].
Responsible Organization 1. HSBC - The software failure incident was caused by a problem at HSBC, affecting payments made by the companies through Britain’s Bacs system [38843, 38695].
Impacted Organization 1. Thousands of Britons who failed to receive their wages [Article 38843, Article 38695] 2. Employers who use HSBC's business banking accounts to make payments [Article 38695] 3. Businesses unable to pay their suppliers due to online banking facilities being temporarily closed [Article 38695]
Software Causes 1. The software cause of the failure incident was a fault in the information used to process payments from HSBC business customers, affecting 275,000 payments including payments to customers of other banks [38695]. 2. The glitch affected payments made by companies through Britain’s Bacs system, which processes electronic transactions, indicating a software issue within the Bacs system [38843].
Non-software Causes 1. High volume of payments scheduled on the Friday before a bank holiday weekend, leading to increased pressure on the banking system [38843, 38695] 2. Fault in the information used to process payments from HSBC business customers [38695]
Impacts 1. Thousands of Britons failed to receive their wages on the Friday before a bank holiday weekend due to the software failure incident at HSBC [38843, 38695]. 2. Employers who use HSBC's business banking accounts were unable to make payments, leaving potentially hundreds of thousands of people without their pay [38695]. 3. Businesses faced challenges in paying their suppliers as the online banking facilities were temporarily closed [38695]. 4. Individuals had to deal with bounced payments, overdraft charges, and financial uncertainties, impacting their plans for the bank holiday weekend and personal events like weddings [38695]. 5. The glitch caused frustration and inconvenience for customers, leading to concerns about financial stability and the reliability of banking systems [38843, 38695].
Preventions 1. Implementing thorough testing procedures before deploying any changes to the electronic payments system could have prevented the software failure incident [38843, 38695]. 2. Investing in upgrading and maintaining robust IT infrastructure and systems could have helped prevent such glitches in the electronic payments system [38843]. 3. Enhancing monitoring and alert systems to detect issues early on and take proactive measures to prevent widespread failures could have mitigated the impact of the software failure incident [38695].
Fixes 1. Implementing immediate steps to ensure the affected payments reach beneficiaries as quickly as possible [38695]. 2. Working with other banks to ensure customers do not lose out as a result of the IT failure [38695]. 3. Investing in the improvement and upgrade of the IT payments systems to prevent future failures [38843].
References 1. HSBC bank statement - The articles gather information about the software failure incident from statements provided by HSBC regarding the glitch affecting payments made by the companies through Britain’s Bacs system [Article 38843, Article 38695]. 2. Affected individuals and businesses - Information is gathered from individuals and businesses affected by the IT glitch at HSBC, such as business owners, employees, and customers [Article 38695]. 3. Social media - The articles mention individuals affected by the glitch venting their frustrations on social media platforms [Article 38843]. 4. Lawmaker - Conservative lawmaker Andrew Tyrie, who chairs parliament’s Treasury Committee, is quoted expressing concerns about the IT failures in major clearing banks [Article 38843]. 5. Various affected individuals - The articles provide insights from various individuals affected by the glitch, such as business owners, employees, and customers, who shared their experiences and concerns [Article 38695].

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: - HSBC has experienced similar IT failures in the past. State-backed Royal Bank of Scotland (RBS) has also faced technology failures, promising to invest in its computer systems after a glitch in June [38843]. - HSBC itself has faced previous technology failures. The recent incident with the electronic payments system affecting salary payments is another example of such failures [38695]. (b) The software failure incident having happened again at multiple_organization: - The article mentions that Britain’s retail banks, in general, have been hit by technology failures causing inconvenience for customers. This indicates that multiple organizations in the banking sector have faced similar IT issues [38843]. - The incident at HSBC is not isolated, as other banks have also faced IT failures. The article mentions that RBS has been the worst affected by technology failures, indicating that multiple organizations in the banking sector have experienced similar issues [38843].
Phase (Design/Operation) design, operation (a) The software failure incident at HSBC, where thousands of Britons failed to receive their wages, was primarily due to a glitch affecting payments made by companies through Britain’s Bacs system [38843, 38695]. This glitch was a result of a fault in the information used to process payments from HSBC business customers [38695]. The incident highlights a failure in the design phase, where contributing factors introduced during system development or updates led to the issue in processing electronic transactions. (b) The software failure incident also impacted the operation of the system, as individuals and businesses faced challenges such as bounced payments, overdraft charges, and the inability to make payments to suppliers due to the IT glitch at HSBC [38695]. This operational failure was a direct result of the system's malfunction during the processing of payments, affecting the day-to-day operations of both customers and businesses relying on the banking services provided by HSBC.
Boundary (Internal/External) within_system (a) within_system: The software failure incident at HSBC was due to a fault in the information used to process some payments from HSBC business customers [38695]. The glitch affected payments made by the companies through Britain’s Bacs system, which processes electronic transactions [38843]. HSBC mentioned that they are taking immediate steps to ensure the payments reach beneficiaries as quickly as possible and will work with other banks to ensure customers do not lose out as a result of the problem [38695]. (b) outside_system: The software failure incident at HSBC was not explicitly attributed to factors originating from outside the system in the provided articles.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident at HSBC was primarily due to non-human actions. The incident was caused by a fault in the information used to process payments from HSBC business customers, affecting 275,000 payments including those to customers of other banks [38695]. The glitch affected payments made through Britain’s Bacs system, which processes electronic transactions [38843]. The bank mentioned taking immediate steps to ensure the payments reach beneficiaries as quickly as possible and working with other banks to prevent customers from losing out due to the problem [38695]. (b) Human actions also played a role in the software failure incident. Employers who use HSBC's business banking accounts were unable to make payments, leading to thousands of people being left without their salaries [38695]. The incident caused significant distress to individuals who were relying on their pay before the bank holiday weekend, with some having to go over their overdrafts and facing financial difficulties [38695]. Customers expressed frustration with the lack of clear communication and resolution timelines from HSBC, indicating a human element in the handling of the situation [38695].
Dimension (Hardware/Software) software (a) The software failure incident reported in the articles is primarily related to software issues rather than hardware. The incident at HSBC was caused by a fault in the information used to process payments from HSBC business customers, affecting electronic transactions made through the Bacs system [38695]. The glitch specifically impacted payments made by companies through the electronic payments system used by HSBC's business banking customers [38843]. Additionally, the bank mentioned that there was a problem with Bacs payments, which are responsible for the electronic processing of financial transactions within the UK [38695]. (b) The software failure incident was attributed to software issues originating from the bank's systems. HSBC acknowledged that there was a fault in the information used to process payments, indicating a software-related problem [38695]. The glitch affected payments made by companies through the Bacs system, highlighting a software issue in the electronic transactions processing [38843]. The bank also mentioned taking immediate steps to ensure payments reach beneficiaries as quickly as possible, indicating a software-related resolution process [38695].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident at HSBC, where thousands of Britons failed to receive their wages due to a glitch affecting payments made by companies through Britain’s Bacs system, does not appear to be malicious. The incident was attributed to a fault in the information used to process payments from HSBC business customers [38843, 38695]. (b) The software failure incident at HSBC was non-malicious, as it was caused by a fault in the information used to process payments from business customers, leading to the failure of 275,000 individual payments [38843, 38695].
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident at HSBC, which resulted in thousands of people not receiving their salaries, can be attributed to poor decisions made in the management of the bank's IT systems. The incident was described as a "serious IT failure" by Conservative lawmaker Andrew Tyrie, highlighting the need for significant investment in the bank's IT infrastructure [38843]. Additionally, the bank's customers expressed frustration over the lack of clear communication and resolution timelines, indicating a lack of proactive decision-making in handling the issue [38695]. These factors point towards poor decisions contributing to the software failure incident.
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident at HSBC, where thousands of Britons failed to receive their wages, can be attributed to development incompetence. The incident was caused by a glitch affecting payments made by companies through Britain’s Bacs system, which processes electronic transactions [38843]. The failure resulted in significant inconvenience for customers, with many expressing frustration on social media and facing financial difficulties due to the delayed payments [38695]. (b) The software failure incident at HSBC, leading to thousands of people being left without their salaries, can also be categorized as accidental. The failure was described as an IT glitch that affected the electronic payments system used by the bank's business banking customers to make payments, including salary payments [38695]. The incident caused a range of problems for employees and businesses, such as bounced payments, overdraft charges, and disrupted plans for the bank holiday weekend [38695].
Duration temporary From the provided articles [38843, 38695], the software failure incident at HSBC causing payment disruptions was temporary in nature. The incident was described as a glitch affecting electronic payments made by the bank's business customers, resulting in the failure of around 275,000 individual payments. The bank assured customers that the majority of transactions would be completed over the course of Friday afternoon and early evening, with any remaining payments completed overnight. HSBC also mentioned taking immediate steps to ensure the affected payments reach beneficiaries as quickly as possible and working with other banks to prevent customers from losing out due to the issue. This indicates that the failure was temporary and efforts were being made to rectify the situation promptly.
Behaviour crash, omission, value, other (a) crash: The software failure incident in the articles can be categorized as a crash as it led to a situation where the system lost its state and failed to perform its intended functions. Thousands of people were left without their salaries due to an IT glitch at HSBC, which prevented employers from making payments, resulting in 275,000 individual payments failing to go through [38695]. (b) omission: The incident can also be classified as an omission, where the system omitted to perform its intended functions at an instance(s). For example, none of the staff in various businesses received their salaries as expected, causing significant financial concerns and disruptions [38695]. (c) timing: While the system was intended to process payments correctly, the timing of the incident caused delays and disruptions. Many individuals were left without funds before a bank holiday weekend, impacting their plans and financial obligations [38843, 38695]. (d) value: The failure can be attributed to a value issue as the system performed its intended functions incorrectly. Payments, including salaries and supplier payments, were affected, causing financial distress and inconvenience to customers [38695]. (e) byzantine: The incident does not exhibit characteristics of a byzantine failure where the system behaves erroneously with inconsistent responses and interactions. The failure primarily resulted in delayed or failed payments without indications of inconsistent behavior [38695, 38843]. (f) other: The other behavior observed in this software failure incident is the inability of the online banking facilities to function properly, leading to error warnings and the temporary closure of online banking facilities for businesses banking with HSBC. This additional aspect of the failure highlights the broader impact on various banking operations beyond just payment processing [38695].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay (a) death: People lost their lives due to the software failure (b) harm: People were physically harmed due to the software failure (c) basic: People's access to food or shelter was impacted because of the software failure (d) property: People's material goods, money, or data was impacted due to the software failure (e) delay: People had to postpone an activity due to the software failure (f) non-human: Non-human entities were impacted due to the software failure (g) no_consequence: There were no real observed consequences of the software failure (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? The software failure incident at HSBC did not result in any direct physical harm, loss of life, or impact on basic needs like food or shelter. However, the incident did lead to significant delays and financial impacts on individuals and businesses. Many people were unable to receive their salaries on time, causing financial stress and inconvenience [38843, 38695]. Additionally, businesses faced challenges in making payments to suppliers, affecting their operations [38695]. The delay in payments also disrupted personal plans, such as wedding arrangements and rental deposits, causing emotional distress and uncertainty [38695]. The consequences were primarily related to financial disruptions and delays rather than physical harm or loss of life.
Domain finance (a) The failed system was related to the finance industry, specifically affecting the processing of electronic transactions for business customers of HSBC [38843, 38695].

Sources

Back to List