Incident: TSB Online Banking Chaos: System Upgrade Failure and Data Breach

Published Date: 2018-04-24

Postmortem Analysis
Timeline 1. The software failure incident at TSB, leading to online banking chaos and issues with account balances and access, happened in April 2018 [71025].
System 1. TSB's online banking system 2. TSB's mobile banking system 3. TSB's digital services 4. TSB's website and app 5. TSB's IT systems 6. TSB's computer systems 7. TSB's online banking service 8. TSB's internet banking services 9. TSB's online and mobile services 10. TSB's platform 11. TSB's data protection measures 12. TSB's banking services These systems and components failed during the software failure incident reported in Article #71025.
Responsible Organization 1. TSB's system upgrade led to the online banking chaos, causing millions of customers to be locked out of their accounts or have wrong balances, as well as allowing access to other customers' accounts and sensitive information [71025]. 2. TSB CEO Paul Pester acknowledged the issues and apologized for the failure, promising that customers wouldn't be left out of pocket [71025]. 3. TSB's social media team faced complaints from customers who reported being able to see other people's accounts after logging back in, indicating a breach in data protection [71025]. 4. The TSB boss, Paul Pester, mentioned that the bank's IT systems could not cope with the demand from customers looking to log on online, leading to the failure [71025].
Impacted Organization 1. TSB customers [71025, 71025] 2. TSB itself [71025]
Software Causes 1. System upgrade led to disaster, causing millions of customers to be locked out of their accounts or have wrong balances [71025]. 2. Issues included being able to access other customers' accounts, view sensitive personal information, and transfer funds due to software glitches [71025]. 3. Customers reported seeing obscene account balances, some owing large amounts, and some being given far more money than they should have in their accounts [71025]. 4. TSB's online and mobile services had to be taken down to resolve the issues, indicating software failures [71025]. 5. The bank experienced intermittent issues with its mobile and internet banking services even after the scheduled upgrade window [71025].
Non-software Causes 1. Insufficient bandwidth in the platform to cope with the number of users logging on [71025] 2. Failure to act quickly enough to address the issues [71025] 3. Lack of detailed and honest statements regarding the incident [71025] 4. Inadequate response to customer complaints and concerns [71025]
Impacts 1. Customers were locked out of their accounts or had wrong balances, with some seeing obscene account balances ranging from $83,000 to over $13,000,000 [71025]. 2. Customers reported being able to access other customers' accounts, view sensitive personal information, and even transfer funds [71025]. 3. Some customers were unable to make bill payments or withdraw cash due to being locked out of their accounts [71025]. 4. The software failure incident led to a breach of data protection, with customers being able to see other people's accounts, including account numbers, sort codes, and transaction histories [71025]. 5. Customers faced difficulties contacting TSB about the issues, with reports of being unable to get through or being put on hold for hours [71025]. 6. The incident caused financial distress and inconvenience to customers, with some having incorrect balances, being locked out of their accounts, and facing challenges in making payments or accessing their money [71025].
Preventions 1. Proper testing and quality assurance before implementing the system upgrade could have prevented the software failure incident [71025]. 2. Ensuring robust data protection measures and security protocols to prevent unauthorized access to customer accounts could have mitigated the impact of the incident [71025]. 3. Prompt and transparent communication with customers about the issues and steps being taken to resolve them could have helped manage customer expectations and reduce frustration [71025].
Fixes 1. Implement a thorough system upgrade testing process to ensure that system upgrades do not lead to disastrous consequences like the one experienced by TSB [71025]. 2. Enhance data protection measures to prevent customers from accessing other customers' accounts and sensitive personal information [71025]. 3. Improve customer communication and support channels to address customer concerns promptly and effectively, especially during system failures [71025]. 4. Conduct a comprehensive review of the system's capacity and bandwidth to ensure it can handle the number of users accessing online services without issues [71025].
References 1. TSB's CEO, Paul Pester 2. TSB customers 3. Sky News 4. Mail Online 5. Philip Augar, former TSB board member 6. TSB website 7. TSB social media team 8. Twitter users 9. BBC 10. Lee MacDonald, former Grange Hill star 11. Judi Rudd, dog trainer from Manchester

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident having happened again at one_organization: The TSB online banking chaos incident reported in Article 71025 is an example of a software failure incident that happened within the same organization. This incident involved millions of customers being locked out of their accounts, wrong balances being displayed, access to other customers' accounts, and issues with online and mobile services after a system upgrade. TSB's CEO apologized for the chaos and promised compensation to affected customers, indicating a significant failure within the organization's software systems [71025]. (b) The software failure incident having happened again at multiple_organization: There is no specific information in the provided articles about a similar software failure incident happening again at other organizations or with their products and services.
Phase (Design/Operation) design, operation (a) The software failure incident in Article 71025 was primarily due to issues related to the design phase of the system. The chaos and problems faced by TSB's online banking system were a result of a system upgrade that led to disaster, causing millions of customers to be locked out of their accounts or have incorrect balances. Customers reported seeing obscene account balances, being able to access other customers' accounts, view sensitive personal information, and even transfer funds. These issues were a direct result of the system upgrade and the associated design flaws introduced during the development process [71025]. (b) Additionally, the software failure incident in Article 71025 also involved failures related to the operation phase of the system. Customers faced difficulties in accessing their accounts, making bill payments, or withdrawing cash due to the operational issues caused by the system upgrade. Many customers reported being unable to contact TSB for assistance, being put on hold for hours, and experiencing delays in resolving their account access problems. These operational challenges contributed to the overall chaos and disruption experienced by TSB's customers [71025].
Boundary (Internal/External) within_system, outside_system (a) The software failure incident reported in the articles is primarily within_system. The failure was caused by a system upgrade that led to disastrous consequences for TSB's online banking services. Customers experienced issues such as being locked out of their accounts, seeing incorrect balances, accessing other customers' accounts, and being unable to make transactions or withdraw cash ([71025]). The CEO of TSB, Paul Pester, acknowledged the failures within the system and apologized for the chaos caused by the online banking issues. He mentioned that the bank's computer systems were not powerful enough to cope with the numbers of users logging on, leading to the inability to access online accounts ([71025]). (b) Additionally, there were contributing factors outside the system that exacerbated the software failure incident. For example, customers faced challenges in reaching TSB for support, with reports of being unable to get through or being put on hold for hours when trying to contact the bank about the issues ([71025]). The external impact of the failure is evident in the experiences of customers like Lee MacDonald, whose locksmith business was paralyzed due to the reliance on TSB's online banking service, leading to the inability to pay staff or suppliers and resulting in canceled jobs and deliveries ([71025]).
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident in Article 71025 was primarily due to non-human actions. The failure occurred after a system upgrade led to disaster, causing millions of customers to be locked out of their accounts or have incorrect balances. Issues included customers being able to access other customers' accounts, view sensitive personal information, and even transfer funds. The CEO mentioned that the online and mobile services needed to be taken down to resolve the issues, indicating a technical problem rather than a human error [71025]. (b) However, human actions also played a role in exacerbating the software failure incident. Customers reported seeing obscene account balances, some owing large amounts of money, and being locked out of their accounts for days. Additionally, the CEO faced criticism for his handling of the crisis, resorting to jargon to explain the IT meltdown and facing ridicule for claiming everything was running smoothly for most customers. Customers on Twitter accused the bank of failing to act quickly enough in response to the issues, highlighting potential shortcomings in human decision-making and communication during the incident [71025].
Dimension (Hardware/Software) hardware, software (a) The software failure incident occurring due to hardware: - The TSB online banking chaos was caused by a system upgrade that led to disaster, resulting in millions of customers being locked out of their accounts or having wrong balances [71025]. - Customers reported seeing obscene account balances, some owing large amounts, such as $83,000 to $13,000,000, even though their accounts are in pounds [71025]. - Issues included being able to access other customers' accounts, view sensitive personal information, and transfer funds due to the system upgrade [71025]. - The upgrade window was scheduled between Friday at 4 pm and Sunday at 6 pm, but there were still 'intermittent issues' with mobile and internet banking services [71025]. (b) The software failure incident occurring due to software: - The TSB CEO mentioned that the online and mobile services would be taken down to resolve the issues, indicating a software-related problem [71025]. - Customers faced issues with access to digital services, leading to the decision to take the mobile app and online banking down for a few hours to address the problems [71025]. - Customers reported being unable to access their money, make payments, or withdraw cash due to the software issues [71025]. - The TSB social media team received complaints from customers about being able to see other people's accounts after logging back in, highlighting a software-related breach [71025].
Objective (Malicious/Non-malicious) malicious, non-malicious From the provided articles, the software failure incident at TSB can be categorized as both malicious and non-malicious: (a) Malicious: - Customers reported being able to access other customers' accounts, view sensitive personal information, and even transfer funds, indicating a potential breach of data security [71025]. - Some customers saw obscene account balances, with amounts ranging from $83,000 to over $13,000,000, which could be considered as a malicious act to disrupt the system [71025]. (b) Non-malicious: - The incident was primarily attributed to a system upgrade that led to chaos, with millions of customers locked out of their accounts or experiencing wrong balances [71025]. - CEO Paul Pester mentioned that the issues were specific to the digital channels and were related to the platform's bandwidth not being able to cope with the number of users, rather than a data breach [71025].
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident at TSB was primarily due to poor_decisions. The chaos in the online banking system was a result of a system upgrade that led to disastrous consequences, including customers being locked out of their accounts, seeing incorrect balances, accessing other customers' accounts, and facing issues with transferring funds [71025]. The CEO, Paul Pester, faced criticism for the bank's response to the crisis, and customers expressed outrage over the situation, highlighting the poor decisions made in handling the system upgrade and its aftermath. Additionally, the CEO's explanation about the IT meltdown and the inability of the bank's computers to cope with the demand from customers logging on online further indicates poor decision-making in system management [71025].
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident in Article 71025 was primarily due to development incompetence. TSB's online banking chaos, where customers were locked out of their accounts, saw wrong balances, and were able to access other customers' accounts, was a result of a system upgrade that led to disaster. CEO Paul Pester apologized for the chaos and mentioned that the level of service provided was not up to the standard expected from TSB [71025]. (b) The software failure incident in Article 71025 also had elements of accidental factors contributing to the failure. Customers reported seeing obscene account balances, being able to view sensitive personal information of other customers, and even transfer funds. These issues were not intentional but occurred due to the system upgrade causing unexpected glitches and faults in the online banking system [71025].
Duration temporary The software failure incident reported in Article 71025 was temporary. The incident involved a system upgrade that led to chaos in TSB's online banking services, causing millions of customers to be locked out of their accounts or have incorrect balances for several days. The CEO mentioned that the online and mobile services would be taken down temporarily to address the issues and hopefully be back up later in the afternoon. Customers reported being unable to access their accounts, seeing incorrect balances, accessing other customers' accounts, and facing difficulties in making payments or withdrawing cash. The bank acknowledged the issues and apologized to customers for the disruption caused by the software failure incident [71025].
Behaviour crash, omission, value, other (a) crash: The software failure incident in Article 71025 resulted in a crash as customers were locked out of their accounts, had wrong balances, and were unable to access online and mobile services after a system upgrade led to disaster [71025]. (b) omission: The software failure incident in Article 71025 involved omission as customers reported issues such as being unable to access their accounts, seeing wrong balances, and facing difficulties in making bill payments or withdrawing cash [71025]. (c) timing: The software failure incident in Article 71025 did not specifically mention timing-related failures. (d) value: The software failure incident in Article 71025 included a value-related failure where customers saw incorrect and obscene account balances, some owing large amounts of money, and others being credited with more money than they should have had in their accounts [71025]. (e) byzantine: The software failure incident in Article 71025 did not exhibit byzantine behavior. (f) other: The software failure incident in Article 71025 also involved customers being able to access other customers' accounts, view sensitive personal information, and even transfer funds, indicating a breach of data protection and privacy [71025].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay (d) property: People's material goods, money, or data was impacted due to the software failure - The software failure incident at TSB led to customers experiencing issues such as wrong balances in their accounts, with some seeing obscene account balances ranging from $83,000 to more than $13,000,000 [71025]. - Customers reported being able to access other customers' accounts, view sensitive personal information, and even transfer funds due to the software failure [71025]. - Some customers found themselves locked out of their accounts for days, unable to pay bills or withdraw cash from machines [71025]. - One customer mentioned being able to see accounts belonging to someone else, including a savings account, an Isa, and a business account, with access to account numbers, sort codes, and transaction histories [71025]. - Another customer discovered they had been credited with £13,000 after logging back in, indicating discrepancies in account balances due to the software failure [71025]. - A former child actor running a locksmith business mentioned that his firm was entirely reliant on TSB's online banking service, which was paralyzed by the IT collapse, impacting his ability to pay staff and suppliers [71025].
Domain finance (a) The failed system was related to the finance industry, specifically online banking services provided by TSB. The incident led to chaos in online banking, with customers experiencing issues such as being locked out of their accounts, seeing incorrect balances, accessing other customers' accounts, and being unable to make transactions or withdrawals ([71025]).

Sources

Back to List