Incident: RBS and Barclays Online Banking Outage: Firewall Upgrade Failure.

Published Date: 2018-09-21

Postmortem Analysis
Timeline 1. The software failure incident involving RBS, NatWest, and Ulster Bank customers happened on September 21, 2018 [75431, 76240].
System 1. Firewall software upgrade at RBS [75431, 76240] 2. Online and mobile banking systems at RBS, NatWest, and Ulster Bank [75431, 76240]
Responsible Organization 1. RBS - The software failure incident was caused by an upgrade to firewall software at RBS that backfired, leaving customers locked out of their accounts [75431, 76240]. 2. Barclays - Some Barclays customers were also affected by a technical problem that left them struggling to log into their accounts [75431, 76240].
Impacted Organization 1. Royal Bank of Scotland (RBS) customers [75431, 76240] 2. NatWest customers [75431, 76240] 3. Ulster Bank customers [75431, 76240] 4. Barclays customers [75431, 76240]
Software Causes 1. An upgrade to firewall software at RBS backfired, leaving customers locked out of online banking and the mobile app [Article 75431]. 2. A firewall security update carried out overnight at RBS shut down their website and app, causing the systems failure [Article 76240].
Non-software Causes 1. The failure incident at RBS, NatWest, and Ulster Bank was caused by an upgrade to firewall software at RBS that backfired, leaving customers locked out of their accounts [75431]. 2. The outage at RBS, NatWest, and Ulster Bank was due to a firewall security update carried out overnight that shut down their website and app [76240].
Impacts 1. Millions of RBS, NatWest, and Ulster Bank customers were locked out of online banking for five hours, unable to settle bills or access their accounts, causing inconvenience and frustration for customers [Article 76240]. 2. Customers faced issues such as being unable to use cash cards in ATMs, pay bills, transfer money, and access their accounts on payday [Article 76240]. 3. The service failure led to complaints flooding the banks' Twitter feeds and phone lines, with customers expressing anger and disappointment [Article 75431, Article 76240]. 4. The incident raised concerns about the reliability and trustworthiness of online banking services, impacting customer confidence in the banking infrastructure [Article 75431]. 5. The failure resulted in demands for explanations and compensation from the head of the Treasury Committee, Nicky Morgan MP, highlighting the seriousness of the impact on customers [Article 75431].
Preventions 1. Regular and thorough testing of software updates before implementation could have prevented the software failure incident at RBS, NatWest, and Ulster Bank [75431, 76240]. 2. Implementing a more robust and comprehensive backup system to quickly restore services in case of a failure could have mitigated the impact of the software failure incident [75431, 76240]. 3. Enhancing the monitoring and alert systems to quickly identify and address any issues that arise during software updates or changes could have helped prevent the widespread outage experienced by customers [75431, 76240].
Fixes 1. Implement thorough testing procedures before deploying software updates to prevent unforeseen issues [75431, 76240]. 2. Enhance communication channels to promptly inform customers about the status of online banking services during outages [75431, 76240]. 3. Invest in improving cyber resilience and IT infrastructure to prevent future service disruptions [76240]. 4. Provide compensation to customers who faced financial losses or inconvenience due to the software failure [75431, 76240]. 5. Conduct a thorough post-incident analysis to identify root causes and prevent similar incidents in the future [75431, 76240].
References 1. RBS spokesperson [75431] 2. Nicky Morgan, head of the Treasury Committee [75431] 3. Ross McEwan, RBS chief executive [75431, 76240] 4. Jes Staley, Barclays chief executive [75431] 5. Customers on Twitter [75431, 76240] 6. NatWest [76240] 7. Ulster Bank [75431, 76240] 8. The Bank of England and the Financial Conduct Authority [76240] 9. Reuters [76240]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: The Royal Bank of Scotland (RBS) experienced a software failure incident in 2012 due to a software update that resulted in millions of customers being locked out of their accounts [Article 75431]. This incident is reminiscent of the recent technical failures at RBS, NatWest, and Ulster Bank in 2018, where customers were unable to access online and mobile accounts due to a firewall software upgrade that backfired [Article 75431]. (b) The software failure incident having happened again at multiple_organization: In addition to RBS, other banks like Barclays, Co-operative Bank, and Cashplus have also faced online disruptions and technical problems, leading to customer complaints and service outages [Article 75431]. The incident at Barclays involved customers being locked out of their accounts due to a technical problem, similar to the issues faced by RBS customers [Article 76240]. This indicates a broader trend of technical failures affecting multiple organizations in the banking sector.
Phase (Design/Operation) design, operation (a) The software failure incident at RBS, NatWest, and Ulster Bank was attributed to an upgrade to firewall software at RBS that backfired, causing customers to be locked out of their online and mobile accounts [75431, 76240]. This failure was related to contributing factors introduced during the system development phase, specifically during the design phase when the firewall software upgrade was implemented to enhance security but resulted in unintended consequences. (b) The software failure incident also involved issues with the operation of the system. Customers were unable to settle bills, pay rent, use cash cards in ATMs, or access their accounts, indicating operational failures that impacted users' ability to carry out essential banking activities [76240]. This aspect of the failure was related to contributing factors introduced during the operation phase, where the system's functionality was affected, leading to disruptions in customer services.
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident at RBS, NatWest, and Ulster Bank was primarily caused by an upgrade to firewall software at RBS that backfired, leading to customers being locked out of their online and mobile accounts [75431, 76240]. Additionally, a firewall security update carried out overnight by RBS shut down their website and app, causing the systems failure that affected millions of customers [76240]. (b) outside_system: The software failure incident was exacerbated by the fact that customers were unable to settle bills, pay rent, or access their accounts, indicating external consequences of the failure [75431, 76240]. Additionally, the incident led to demands for compensation for customers who suffered losses due to the failure, highlighting the impact on customers beyond just the technical issues [75431].
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident occurring due to non-human actions: - The incident at RBS, NatWest, and Ulster Bank was caused by an upgrade to firewall software at RBS that backfired, leading to customers being locked out of their online and mobile accounts [75431]. - The outage at RBS was due to a firewall security update carried out overnight that shut down their website and app, causing the systems failure [76240]. (b) The software failure incident occurring due to human actions: - The head of the Treasury Committee, Nicky Morgan MP, demanded answers from RBS and Barclays, highlighting the litany of failures of banking IT systems and questioning the banks' justifications for branch closures based on providing seamless online and mobile banking services [75431]. - RBS chief executive Ross McEwan initially couldn't say what the problem was, and customers criticized the bank for the lack of access to their accounts, inability to pay bills, and difficulties in withdrawing cash [76240].
Dimension (Hardware/Software) hardware, software (a) The software failure incident occurring due to hardware: - The incident at RBS, NatWest, and Ulster Bank was caused by an upgrade to firewall software at RBS that backfired, leading to customers being locked out of their accounts [75431]. - An overnight firewall security update at RBS shut down their website and app, causing the systems failure that locked out millions of customers [76240]. (b) The software failure incident occurring due to software: - The failure at RBS, NatWest, and Ulster Bank was primarily attributed to a software upgrade that was intended to enhance security but resulted in customers being unable to access their accounts [75431]. - The IT glitch at RBS, NatWest, and Ulster Bank was caused by a firewall security update carried out overnight that led to the shutdown of their website and app, indicating a software-related issue [76240].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident reported in the articles appears to be non-malicious in nature. The incidents at RBS, NatWest, and Ulster Bank were attributed to technical issues resulting from a firewall software upgrade that backfired, causing customers to be locked out of their online and mobile banking accounts [75431, 76240]. The failures were described as unintended consequences of system upgrades rather than deliberate actions to harm the systems. Additionally, the apologies issued by the bank executives and the efforts to resolve the issues indicate a non-malicious intent behind the failures.
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident at RBS, Barclays, and other banks was primarily due to poor_decisions. The incident was caused by an upgrade to firewall software at RBS that backfired, leaving customers locked out of their accounts [75431]. Additionally, a firewall security update carried out overnight at RBS shut down their website and app, causing the nationwide outage [76240]. The failure was exacerbated by the closure of branches and the push towards online banking services, highlighting the poor decisions made in implementing and managing the IT systems of these banks.
Capability (Incompetence/Accidental) development_incompetence (a) The software failure incident occurring due to development incompetence: - The incident at RBS was caused by an upgrade to firewall software that backfired, leaving customers locked out of their accounts [Article 75431]. - RBS, NatWest, and Ulster Bank customers were locked out of online banking due to a firewall security update carried out overnight that shut down the website and app [Article 76240]. (b) The software failure incident occurring accidentally: - RBS chief executive initially couldn't identify the problem causing the outage, indicating it was an accidental issue [Article 76240]. - Customers complained about being unable to access their accounts or cash from ATMs, suggesting an accidental failure in the system [Article 76240].
Duration temporary (a) The software failure incident described in the articles was temporary. The incident caused millions of RBS, NatWest, and Ulster Bank customers to be locked out of their online banking for a period of five hours [Article 76240]. The outage was due to a firewall security update that was carried out overnight, which inadvertently shut down the website and app, leading to customers being unable to settle bills or access their accounts [Article 76240]. (b) The software failure incident was not permanent as it was resolved within the same day, and customers were eventually able to access their accounts once the issue was fixed [Article 76240].
Behaviour crash, omission, timing, value, other (a) crash: The software failure incident described in the articles can be categorized as a crash. This is evident from the reports of customers being locked out of their online and mobile banking accounts, unable to access their money, settle bills, or pay rent due to the system failure [75431, 76240]. (b) omission: The incident also involved omission as customers complained about not being able to use their cash cards in ATMs, unable to access their accounts, and facing difficulties in paying bills or transferring money [76240]. (c) timing: Timing issues were present in the failure incident as well. Customers expressed frustration over not being able to access their accounts on payday, leading to concerns about paying bills and facing late payment charges [76240]. (d) value: The software failure incident also included value-related failures. Customers were unable to perform transactions correctly, such as transferring money, paying bills, and accessing their accounts for essential financial activities [76240]. (e) byzantine: There is no explicit mention of the software failure incident exhibiting byzantine behavior in the articles. (f) other: The software failure incident can be categorized under the "other" behavior as well. This includes the system update intended to improve security backfiring and causing customers to be locked out of their accounts, leading to disruptions in online and mobile banking services [75431].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay, non-human, theoretical_consequence, other (a) death: There is no mention of any deaths resulting from the software failure incident in the provided articles [75431, 76240]. (b) harm: There is no mention of physical harm to individuals due to the software failure incident in the provided articles [75431, 76240]. (c) basic: People's access to food or shelter was not directly impacted by the software failure incident [75431, 76240]. (d) property: The software failure incident impacted people's access to their accounts, causing inconvenience and potential financial losses [75431, 76240]. (e) delay: Customers experienced delays in accessing their accounts, settling bills, and making transactions due to the software failure incident [75431, 76240]. (f) non-human: Non-human entities, such as banking systems and online platforms, were directly impacted by the software failure incident [75431, 76240]. (g) no_consequence: There were real observed consequences of the software failure incident, including customers being locked out of their accounts and facing difficulties in conducting financial transactions [75431, 76240]. (h) theoretical_consequence: There were potential consequences discussed, such as compensation for customers and the need for banks to strengthen their contingency plans, but these did not occur as direct outcomes of the software failure incident [75431, 76240]. (i) other: The software failure incident led to customers expressing frustration, inability to pay bills, reliance on alternative banking methods like telephone banking, and considering switching banks due to the inconvenience caused [75431, 76240].
Domain finance, government (a) The software failure incident affected the finance industry, specifically online banking services provided by Royal Bank of Scotland (RBS), NatWest, Ulster Bank, Barclays, and other banking institutions [75431, 76240]. (h) The failure impacted the finance industry by causing disruptions in online and mobile banking services, leading to customers being locked out of their accounts, unable to settle bills, pay rent, access cash, or make transfers [75431, 76240]. (l) The incident also relates to the government sector as the head of the powerful Treasury Committee, Nicky Morgan MP, demanded answers and compensation for customers affected by the technical failures in the banking IT systems [75431, 76240]. (m) The software failure incident is not related to any other industry mentioned in the options provided.

Sources

Back to List