Incident: RBS Group's Repeated Banking System Failures Causing Customer Disruptions

Published Date: 2013-12-03

Postmortem Analysis
Timeline 1. The software failure incident at RBS, NatWest, and Ulster Bank, where customers were locked out of their accounts for days, happened in June 2012 [38697]. 2. Less than a year later, in March 2013, NatWest's systems fell over again, leaving customers unable to withdraw cash or make transactions [38697]. 3. A technological banking glitch on one of the busiest online shopping days of the year, affecting millions of shoppers, occurred on 2 December 2013 [38697]. 4. Lloyds and TSB experienced a technological meltdown in January 2014, leading to customers being unable to withdraw money or use their cards [38697]. 5. RBS suffered another IT fiasco in June 2015, where customers faced delays in receiving payments [38697]. 6. HSBC faced IT failures in August 2015, with business customers unable to make payments [38697]. Estimation: - The incident mentioned in the article published on 2013-12-03 occurred on Cyber Monday, one of the busiest days for online shopping before Christmas. Based on this, the incident likely occurred on December 2, 2013.
System 1. RBS, NatWest, and Ulster Bank systems [38697, 38697, 38697] 2. Lloyds and TSB systems [38697] 3. HSBC systems [38697] 4. RBS's technology systems [55692] 5. RBS's payments system [55692]
Responsible Organization 1. RBS Group [38697, 55692] 2. Cyber attackers [38697]
Impacted Organization 1. RBS, NatWest, and Ulster Bank customers [38697, 38697, 38697] 2. Lloyds and TSB customers [38697] 3. HSBC business customers [38697]
Software Causes 1. Neglect of technology investment for decades leading to outdated and complex systems [55692] 2. Technological glitches and failures in the banking systems of RBS, NatWest, Ulster Bank, Lloyds, TSB, and HSBC [38697]
Non-software Causes 1. Neglect of technology investment for decades by Royal Bank of Scotland [55692] 2. Outdated and complex patchwork of systems due to multiple acquisitions by RBS [55692] 3. Lack of sufficient investment in technology by banks in general during the boom years [55692]
Impacts 1. Millions of customers of RBS, NatWest, and Ulster Bank were locked out of their accounts for days, with Ulster Bank customers being affected for weeks [38697]. 2. Customers were temporarily unable to withdraw cash or make transactions [38697]. 3. Millions of shoppers were unable to pay for transactions using their credit or debit cards on one of the busiest online shopping days of the year [38697]. 4. Some customers were prevented from accessing their accounts due to a cyber attack [38697]. 5. Many people were unable to withdraw money or use their cards due to a technological meltdown at Lloyds and TSB [38697]. 6. RBS customers faced delays in receiving 600,000 payments that failed to enter accounts overnight [38697]. 7. HSBC business customers were unable to pay salaries to staff or make payments to suppliers [38697]. 8. RBS customers were unable to withdraw cash or pay for goods due to a system crash, affecting over 1 million customers [55692]. 9. RBS faced potential costs in compensation and suffered setbacks in its recovery efforts from the financial crisis [55692]. 10. The technology glitch raised concerns about the resilience of RBS's outdated systems and lack of investment in technology [55692].
Preventions 1. Proper investment in technology and systems maintenance over the years could have prevented the software failure incidents at RBS, NatWest, Ulster Bank, Lloyds, TSB, and HSBC [55692]. 2. Regular updates and upgrades to ensure the resilience and reliability of the banking systems could have prevented the recurring glitches and crashes [55692]. 3. Implementing robust cybersecurity measures to prevent cyber attacks that can disrupt banking services [38697]. 4. Conducting thorough testing and quality assurance processes before major events like Cyber Monday to identify and address any potential issues in advance [55692]. 5. Learning from past incidents and regulatory fines to proactively address system weaknesses and vulnerabilities [38697].
Fixes 1. Proper investment in technology infrastructure and systems to ensure resilience and reliability [55692]. 2. Regular updates and upgrades to outdated systems to prevent glitches and crashes [55692]. 3. Enhanced cybersecurity measures to protect against cyber attacks [38697]. 4. Improved monitoring and testing of systems to identify and address potential issues before they escalate [38697]. 5. Compliance with regulatory standards and guidelines to ensure the stability and security of banking systems [55692].
References 1. Ross McEwan, CEO of Royal Bank of Scotland [55692] 2. Stephen Hester, former CEO of Royal Bank of Scotland [55692] 3. Andy Haldane, director for financial stability at the Bank of England [55692] 4. Ralph Silva, research firm SRN [55692]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: - RBS Group experienced multiple software failure incidents, including a technological banking glitch in December 2013 [38697], a system crash in December 2013 [38697], and another IT fiasco in June 2015 [38697]. - The CEO of RBS admitted that the bank had neglected its technology for decades, leading to system crashes and failures affecting millions of customers [55692]. (b) The software failure incident having happened again at multiple_organization: - Lloyds and TSB also faced a technological meltdown in January 2014, affecting their customers' ability to withdraw money or use their cards [38697]. - HSBC joined the ranks of IT failures in August 2015, with many business customers unable to make payments as the bank holiday weekend approached [38697].
Phase (Design/Operation) design, operation (a) The software failure incidents mentioned in the articles are primarily related to the design phase. The failures were attributed to the bank's failure to invest properly in its systems over decades, resulting in outdated technology and a complex patchwork of systems after numerous acquisitions [55692]. The incidents involved system crashes, technology glitches, and cyber attacks that affected millions of customers and led to disruptions in services such as online banking, payments, and cash withdrawals [38697, 55692]. (b) Additionally, there are indications of failures related to the operation phase. For example, customers were temporarily unable to withdraw cash or make transactions, use debit cards, or access their accounts due to system failures and cyber attacks [38697, 55692]. These operational failures resulted in inconvenience to customers and financial losses for the banks, highlighting issues with the operation or misuse of the systems.
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incidents reported in the articles are primarily attributed to issues within the systems of the banks themselves. The failures were caused by a lack of proper investment in technology, outdated systems, and a complex patchwork of systems resulting from multiple acquisitions [55692]. The failures affected various aspects of banking operations such as cash withdrawals, online transactions, and payments, indicating internal system issues [55692]. The failures led to customers being unable to access their accounts, make transactions, or use banking services, highlighting internal system vulnerabilities [38697, 55692]. (b) outside_system: While some incidents, like the cyber attack on RBS in December 2013, were caused by external factors [38697], the overall pattern of software failures in the articles points more towards internal system issues rather than external factors. The failures were mainly attributed to the banks' lack of proper investment in technology, outdated systems, and inadequate infrastructure [55692].
Nature (Human/Non-human) non-human_actions (a) The software failure incidents mentioned in the articles were primarily due to non-human actions such as technological glitches, system crashes, cyber attacks, and outdated systems. For example, the articles highlight incidents where RBS, NatWest, Ulster Bank, Lloyds, TSB, and HSBC experienced system failures, technology glitches, and cyber attacks that led to customers being unable to withdraw cash, make transactions, or access their accounts [38697, 55692]. (b) While the articles do not explicitly mention specific software failures caused by human actions, they do emphasize the lack of proper investment in technology, outdated systems, and the failure to invest in IT infrastructure by banks like RBS. The neglect of technology for decades, lack of proper investment in systems, and the complex patchwork of systems after acquisitions are highlighted as contributing factors to the software failures experienced by these banks [55692].
Dimension (Hardware/Software) software (a) The software failure incidents reported in the articles are primarily attributed to software issues rather than hardware failures. For example, incidents such as banking meltdowns, system crashes, technological glitches, and cyber attacks affecting banks like RBS, NatWest, Ulster Bank, Lloyds, TSB, and HSBC are all related to software failures ([38697], [55692]). These failures are described as system crashes, outdated technology, lack of investment in systems, complex patchwork of systems, and technology glitches, indicating software-related issues rather than hardware failures.
Objective (Malicious/Non-malicious) malicious, non-malicious (a) The articles mention instances of both malicious and non-malicious software failure incidents: Malicious: - In December 2013, RBS Group experienced a cyber attack that prevented some customers from accessing their accounts [38697]. - The article from 2013 highlights a system crash at Royal Bank of Scotland that left over 1 million customers unable to withdraw cash or pay for goods, raising questions about the resilience of RBS's technology [55692]. Non-malicious: - Various technological glitches and meltdowns affected banks like RBS, NatWest, Ulster Bank, Lloyds, and TSB, leading to customers being unable to withdraw money, make transactions, or access their accounts [38697]. - The CEO of RBS admitted that the bank had neglected its technology for decades, leading to outdated systems and a lack of investment, which contributed to system crashes and failures [55692].
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incidents mentioned in the articles point towards poor_decisions as the intent behind the failures. The articles highlight how Royal Bank of Scotland (RBS) neglected its technology for decades, failed to invest properly in its systems, and had outdated technology made up of a complex patchwork of systems after numerous acquisitions [55692]. The failures were attributed to a lack of investment in technology, under-investment in IT, and a failure to build robust systems, ultimately leading to multiple system crashes affecting millions of customers [55692]. The bank's former CEO was blamed for under-investing in technology, and the Bank of England highlighted the need for banks to transform their IT and allocate more resources to technology improvements [55692]. These factors indicate that poor decisions and under-investment in technology played a significant role in the software failure incidents reported in the articles.
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incidents reported in the articles are primarily related to development incompetence. The articles highlight how Royal Bank of Scotland (RBS) neglected its technology for decades, with the bank's boss admitting that RBS failed to invest properly in its systems [55692]. The articles also mention how RBS's technology was regarded as outdated and made up of a complex patchwork of systems after dozens of acquisitions, indicating a lack of professional competence in managing and updating the technology infrastructure [55692]. Additionally, the articles discuss how RBS faced multiple IT fiascos and glitches over the years, indicating a recurring pattern of failure due to development incompetence [38697, 55692]. (b) The software failure incidents reported in the articles also include accidental factors. For example, one incident mentioned in the articles was a cyber attack on RBS that prevented some customers from accessing their accounts [38697]. This incident was not directly attributed to development incompetence but rather an external factor that caused the failure accidentally. Additionally, the articles mention how the latest crash at RBS was not related to volume but did not provide details on what had caused the system crash, indicating a potential accidental element in the failure [55692].
Duration permanent, temporary (a) The articles provide information about both permanent and temporary software failure incidents: 1. Permanent Failure: - The article [55692] mentions a system crash that left more than 1 million customers unable to withdraw cash or pay for goods. This incident lasted for three hours on one of the busiest online shopping days of the year. - It is stated that RBS neglected its technology for decades, leading to a complex patchwork of outdated systems after numerous acquisitions. The bank's chief executive admitted to the lack of proper investment in systems over the years. - The article highlights a more serious crash in RBS's payments system the previous year, which the regulator was still investigating. This indicates a recurring issue with the bank's technology infrastructure. 2. Temporary Failure: - The articles [38697] mention incidents where customers were temporarily unable to withdraw cash, make transactions, or access their accounts due to technological glitches or cyber attacks. These incidents affected millions of customers but were resolved within a certain period. - For example, NatWest's systems fell over temporarily, leaving customers unable to withdraw cash or make transactions. Similarly, a cyber attack on RBS prevented some customers from accessing their accounts, but this was resolved after a few days. Overall, the articles provide evidence of both permanent and temporary software failure incidents within the banking sector, highlighting the ongoing challenges faced by these institutions in maintaining reliable and resilient technology systems.
Behaviour crash, omission, timing, value, other (a) crash: The articles mention instances where the banking systems of RBS, NatWest, and Ulster Bank experienced crashes, leading to customers being unable to withdraw cash, make transactions, or access their accounts [38697, 55692]. (b) omission: The software failures resulted in the omission of performing intended functions, such as customers being temporarily unable to withdraw cash, make transactions, or access their accounts [38697, 55692]. (c) timing: There are indications of timing-related failures, where the systems performed their intended functions incorrectly at critical times, such as on one of the busiest online shopping days of the year, causing inconvenience to customers [55692]. (d) value: The software failures led to the systems performing their intended functions incorrectly, such as customers being unable to pay for transactions using their credit or debit cards [38697]. (e) byzantine: There is no specific mention of the software failures exhibiting byzantine behavior in the articles. (f) other: The software failures also resulted in inconsistent responses and interactions with customers, causing frustration and financial inconvenience [55692].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay, non-human, theoretical_consequence (a) death: There is no mention of any deaths resulting from the software failure incidents reported in the articles. (b) harm: There is no mention of physical harm to individuals due to the software failure incidents. (c) basic: There is no mention of people's access to food or shelter being impacted by the software failure incidents. (d) property: The software failure incidents impacted people's material goods, money, or data. For example, in the incident involving RBS in 2012, customers were unable to withdraw cash or make transactions, potentially impacting their financial transactions [38697]. Additionally, in the incident involving RBS in 2013, customers were unable to pay for transactions using their credit or debit cards [38697]. (e) delay: People had to postpone activities due to the software failure incidents. For instance, in the incident involving RBS in 2013, customers were temporarily unable to withdraw cash or make transactions, impacting their financial activities [38697]. (f) non-human: Non-human entities were impacted by the software failure incidents. For example, in the incident involving RBS in 2013, the bank's technology glitch affected its cash machines, online and mobile banking services, causing inconvenience to customers [55692]. (g) no_consequence: There were observed consequences of the software failure incidents, such as customers being unable to carry out financial transactions or access their accounts. (h) theoretical_consequence: There were potential consequences discussed regarding the software failure incidents. For example, the incidents raised concerns about the resilience of RBS's technology, the need for increased investment in systems, and the potential cost of compensation to affected customers [55692]. (i) other: There were no other consequences mentioned in the articles beyond those related to property, delay, non-human impacts, observed consequences, and theoretical consequences.
Domain finance (a) The failed system was related to the finance industry, specifically affecting banking operations. The incidents involved major banks like RBS, NatWest, Ulster Bank, Lloyds, TSB, and HSBC, causing disruptions in online banking, payments, cash withdrawals, and transactions [38697, 55692]. (h) The software failure incidents were directly impacting the finance industry, as mentioned in the articles. The failures affected millions of customers, leading to issues with accessing accounts, making transactions, and withdrawing money [38697, 55692].

Sources

Back to List