Incident: RBS IT Glitch Impacts Customers' Card Transactions and Payments

Published Date: 2014-11-21

Postmortem Analysis
Timeline 1. The software failure incident at Royal Bank of Scotland (RBS) occurred in 2012, as mentioned in the article [31502]. Therefore, the software failure incident happened in 2012.
System 1. RBS, NatWest, and Ulster Visa debit and credit cards system [31502] 2. Software that processed updates to customers' accounts overnight [31502]
Responsible Organization 1. RBS bosses admitted they had failed to invest enough in their IT systems for decades, following a computer meltdown that hit millions of customers [31502]. 2. In 2012, a junior technician in India was blamed for causing the RBS computer meltdown which froze millions of British bank accounts [31502].
Impacted Organization 1. Royal Bank of Scotland customers [31502] 2. NatWest customers [31502] 3. Ulster Bank customers [31502]
Software Causes 1. The software causes of the failure incident included a glitch that left customers unable to use their credit or debit cards at store checkouts and ATMs [31502]. 2. The failure incident was also attributed to issues with higher-value payments or transactions abroad, affecting a small number of customers [31502]. 3. The failure incident in 2012 was due to a botched upgrade to the software that processed updates to customers' accounts overnight, leading to disruptions in online banking facilities, accurate account balances, delayed mortgage payments, and cash unavailability in foreign countries [31502].
Non-software Causes 1. RBS admitted to failing to invest properly in its IT systems for decades, leading to weaknesses in their computer systems [31502]. 2. The issues in summer 2012 stemmed from a botched upgrade to the software that processed updates to customers' accounts overnight, which was not properly tested before implementation [31502]. 3. The bank's central IT function decided to uninstall the problematic upgrade without testing the consequences, exacerbating the situation [31502]. 4. RBS suffered from another systems outage in December on a busy online shopping day due to undisclosed reasons [31502].
Impacts 1. Customers were unable to use their credit or debit cards at store checkouts and ATMs, affecting those with sister banks NatWest and Ulster Bank [31502]. 2. Higher-value payments or transactions abroad were particularly affected, resulting in minimal customer impact [31502]. 3. Customers experienced frustration and vented their anger on social media platforms like Twitter, criticizing RBS for the glitch [31502]. 4. Some customers reported failed payments and transactions, causing inconvenience and financial issues [31502]. 5. The software failure incident led to disruptions lasting several weeks for around 6.5 million customers, equivalent to 10% of the UK population, in 2012 [31502]. 6. Mortgage payments were delayed, and customers were left without cash in foreign countries due to the IT meltdown in 2012 [31502]. 7. The incident in 2012 stemmed from a botched upgrade to the software that processed updates to customers' accounts overnight, leading to widespread issues [31502].
Preventions 1. Proper investment in IT systems: RBS admitted to failing to invest properly in their IT systems for decades, leading to weaknesses and vulnerabilities [31502]. 2. Thorough testing before actions: The bank uninstalled a botched software upgrade without testing the consequences, which exacerbated the issues [31502]. 3. Improved monitoring and response mechanisms: Quick detection and resolution of issues could have minimized customer impact [31502]. 4. Enhanced cybersecurity measures: Given the increasing frequency of IT failures and hacks, bolstering cybersecurity could have prevented incidents like the 2012 IT meltdown and subsequent glitches [31502].
Fixes 1. Invest more in IT systems to increase resilience and reliability [31502] 2. Conduct thorough testing before implementing upgrades or changes to software [31502] 3. Address any underlying weaknesses in the IT infrastructure that have been neglected for decades [31502] 4. Improve monitoring and response mechanisms to quickly resolve issues when they arise [31502]
References 1. RBS spokesman 2. RBS statement 3. RBS chairman Sir Philip Hampton 4. Former chief executive Stephen Hester 5. RBS chief executive Ross McEwan 6. David Buik from broker Panmure Gordon 7. Regulators 8. UK and US regulators 9. Transcripts of RBS traders 10. Internal investigation by RBS [31502]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident happened again at Royal Bank of Scotland (RBS). The incident in the article describes a glitch that left RBS customers unable to use their credit or debit cards at store checkouts and ATMs, which resulted in frustration and complaints from customers [31502]. (b) The software failure incident has also happened at other organizations. The article mentions that the IT meltdown fine for RBS came just days after the bank was one of six banks fined for rigging the foreign exchange market. Additionally, RBS suffered another systems outage in December on the busiest online shopping day of the year, indicating recurring issues with their IT systems [31502].
Phase (Design/Operation) design (a) The software failure incident related to the design phase can be attributed to the IT meltdown in 2012 at RBS. The incident was caused by a botched upgrade to the software that processed updates to customers' accounts overnight. When problems were noticed with the upgrade, the bank's central IT function decided to uninstall it without first testing the consequences of that action, leading to disruptions for around 6.5 million customers [31502]. (b) The software failure incident related to the operation phase occurred when RBS customers were left unable to use their credit or debit cards at store checkouts and ATMs due to a glitch. The issue affected those attempting to make higher-value payments or transactions abroad, resulting in frustration and complaints from customers. The problem started at 7.45am and was resolved by 9.15am, with RBS stating that only a small number of customers experienced issues and that there was minimal customer impact [31502].
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident at RBS in 2012 was attributed to "unacceptable weaknesses" in its computer systems, with disruptions lasting for several weeks affecting around 6.5 million customers. The issues stemmed from a botched upgrade to the software that processed updates to customers' accounts overnight, and the decision to uninstall it without testing the consequences [31502]. (b) outside_system: The failure in 2012 was also linked to decades of underinvestment in IT systems by RBS, with the bank admitting to failing to invest properly in its systems for years. Additionally, the incident in 2012 was not directly related to the recent glitch that occurred in 2014, indicating that the contributing factors for the 2014 incident were likely external to the system [31502].
Nature (Human/Non-human) non-human_actions (a) The software failure incident was attributed to non-human actions, specifically a glitch in the system that left customers unable to use their credit or debit cards at store checkouts and ATMs [31502]. (b) The articles do not provide specific information attributing the software failure incident to human actions.
Dimension (Hardware/Software) hardware, software (a) The software failure incident occurring due to hardware: - The article mentions that the 2012 IT meltdown at RBS was caused by a botched upgrade to the software that processed updates to customers' accounts overnight. When problems were noticed with the upgrade, the central IT function decided to uninstall it without testing the consequences, leading to disruptions in services [31502]. - It was reported that in 2012, a junior technician in India was initially blamed for causing the RBS computer meltdown that froze millions of British bank accounts. However, it later emerged that the issue originated from their Edinburgh headquarters [31502]. (b) The software failure incident occurring due to software: - The article highlights that RBS admitted to 'unacceptable weaknesses' in its computer systems, indicating that the failure was due to shortcomings in the software systems [31502]. - The bank's chief executive acknowledged that the most recent systems failure was 'unacceptable' and attributed it to decades of underinvestment in their IT systems, indicating a software-related issue [31502].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident at Royal Bank of Scotland (RBS) in 2012 was non-malicious. The incident was attributed to decades of underinvestment in IT systems, with RBS bosses admitting that they had failed to invest enough in their IT systems for years [31502]. The failure stemmed from a botched upgrade to the software that processed updates to customers' accounts overnight, and the decision to uninstall it without testing the consequences first. This led to disruptions for around 6.5 million customers, impacting services like online banking, accurate account balances, mortgage payments, and cash withdrawals [31502]. (b) The article does not mention any malicious intent behind the software failure incident at RBS in 2012.
Intent (Poor/Accidental Decisions) poor_decisions, accidental_decisions (a) The software failure incident at RBS in 2012 was primarily attributed to poor decisions made by the bank regarding their IT systems. RBS admitted to "unacceptable weaknesses" in its computer systems, acknowledging that they had failed to invest properly in their IT systems for decades [31502]. The issues stemmed from a botched upgrade to the software that processed updates to customers' accounts overnight, and when problems were noticed, the decision was made to uninstall it without testing the consequences first. Additionally, RBS chairman Sir Philip Hampton mentioned that the bank had spent hundreds of millions of pounds on increasing the resilience of its IT systems since the incident [31502]. (b) The software failure incident in 2012 at RBS was also partially attributed to accidental decisions or mistakes. For example, in 2012, a junior technician in India was initially blamed for causing the RBS computer meltdown, which froze millions of British bank accounts. However, it later emerged that the issue originated from their Edinburgh headquarters [31502]. This indicates that there were unintended consequences resulting from the decisions made during the incident.
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident at Royal Bank of Scotland (RBS) in 2012 was attributed to development incompetence. RBS admitted to "unacceptable weaknesses" in its computer systems, acknowledging that they had failed to invest properly in their IT systems for decades [31502]. The issues stemmed from a botched upgrade to the software that processed updates to customers' accounts overnight, and the decision to uninstall it without testing the consequences first. The failure was a result of inadequate investment in IT systems and lack of professional competence in managing the software upgrades. (b) The software failure incident at RBS in 2012 was also accidental in nature. The bank's central IT function decided to uninstall the problematic software upgrade without testing the consequences, leading to disruptions for around 6.5 million customers [31502]. This accidental decision exacerbated the IT meltdown, causing issues such as blocked debit card cash withdrawals, delayed mortgage payments, and inaccurate account balances. The accidental actions taken during the software upgrade process contributed to the widespread failure experienced by RBS customers.
Duration temporary (a) The software failure incident mentioned in the articles was temporary. The glitch that left RBS, NatWest, and Ulster Bank customers unable to use their credit or debit cards at store checkouts and ATMs lasted from 7.45am to 9.15am [31502]. This indicates that the failure was not permanent but rather a temporary issue that was quickly resolved.
Behaviour crash, omission, timing, value (a) crash: The software failure incident mentioned in Article 31502 resulted in a crash where customers were unable to use their credit or debit cards at store checkouts and ATMs. The incident caused a disruption in the system's functionality, leading to a loss of service for customers [31502]. (b) omission: The software failure incident also involved omission, as some customers experienced issues with payments failing to be processed. For example, a customer complained that a payment made at Tesco did not go through, indicating an omission in the system's processing of transactions [31502]. (c) timing: The timing of the software failure incident was crucial, as it occurred early in the morning, affecting customers trying to make transactions. The problems started at 7.45 am and were resolved by 9.15 am, indicating that the system's performance was delayed, impacting customers during that time frame [31502]. (d) value: The software failure incident also involved issues related to the value of transactions. The glitch affected customers attempting to make higher-value payments or transactions abroad, resulting in incorrect processing or denial of these transactions. This incorrect behavior of the system led to a disruption in the intended value of the transactions [31502]. (e) byzantine: The software failure incident did not exhibit behaviors related to a byzantine failure, where the system behaves erroneously with inconsistent responses and interactions. The incident primarily involved disruptions in functionality, timing, and value of transactions, rather than exhibiting inconsistent or conflicting behaviors [31502]. (f) other: The software failure incident in the article did not explicitly mention any other specific behavior beyond crash, omission, timing, and value-related issues. The primary focus was on the system's failure to process transactions correctly, leading to customer dissatisfaction and disruptions in service [31502].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay The consequence of the software failure incident described in the articles is primarily related to option (d) property. The software failure incident led to customers being unable to use their credit or debit cards at store checkouts and ATMs, affecting their ability to make payments and transactions [31502]. Additionally, mortgage payments were delayed, and customers were left without cash in foreign countries due to the disruption caused by the IT meltdown in 2012 [31502]. The failure also impacted online banking facilities, accurate account balances from ATMs, and the processing of debit card cash withdrawals and purchases [31502].
Domain finance (a) The failed system was related to the finance industry, specifically affecting customers of Royal Bank of Scotland, NatWest, and Ulster Bank [31502]. The incident involved a glitch that left customers unable to use their credit or debit cards at store checkouts and ATMs, impacting their ability to make payments and transactions [31502]. (h) The incident was directly related to the finance industry, as it affected the banking services provided by Royal Bank of Scotland, NatWest, and Ulster Bank [31502]. The glitch hindered customers' ability to conduct financial transactions using their Visa debit and credit cards [31502]. (m) The incident was not related to an industry outside of the options provided.

Sources

Back to List