Incident: Telstra Fiber Optic Cable Outage Causes Nationwide Payment Disruptions

Published Date: 2019-07-11

Postmortem Analysis
Timeline 1. The software failure incident happened on July 11, 2019. [Article 86875]
System 1. Telstra's network services [86875] 2. EFTPOS systems 3. ATMs
Responsible Organization 1. Telstra [86875] 2. Commonwealth Bank (CBA) [86875] 3. ANZ Bank [86875]
Impacted Organization 1. Commonwealth Bank [86875] 2. ANZ Bank [86875] 3. Woolworths [86875] 4. Australia Post [86875] 5. Xbox [86875] 6. Caltex [86875] 7. NSW Ambulance [86875]
Software Causes 1. Unknown
Non-software Causes 1. Severed fibre optic cable in Sydney [86875]
Impacts 1. Thousands of ATMs were down across Australia, causing chaos and inconvenience to customers [86875]. 2. Payments were rejected at major retailers, affecting transactions for millions of customers [86875]. 3. EFTPOS services were disrupted, leading to issues with electronic payments [86875]. 4. Commonwealth Bank, ANZ, NAB, and Westpac ATMs crashed, impacting banking services for customers [86875]. 5. Other organizations such as Xbox, Woolworths, Australia Post, Caltex, and NSW Ambulance also experienced problems with their Telstra services [86875].
Preventions 1. Implementing redundancy in critical infrastructure components such as fibre optic cables to prevent single points of failure [86875]. 2. Conducting regular maintenance and inspections of the network infrastructure to identify and address potential issues before they lead to widespread outages [86875]. 3. Establishing robust communication and coordination protocols between telecommunication providers and their clients to ensure prompt resolution of issues and minimize service disruptions [86875].
Fixes 1. Repairing the severed fibre optic cable in Sydney could fix the software failure incident [Article 86875].
References 1. Telstra - Telstra is the main entity providing information about the software failure incident as it is their network that experienced the fault [Article 86875]. 2. Commonwealth Bank (CBA) - CBA confirmed the outage and provided updates to customers about the impact on their services [Article 86875]. 3. ANZ Bank - ANZ Bank also confirmed the issues with their services and communicated with their customers about the problem [Article 86875]. 4. Woolworths - Woolworths confirmed they were experiencing problems processing payments due to the Telstra network issue [Article 86875]. 5. Australia Post - Australia Post reported software issues in retail and the impact on their services due to the Telstra fault [Article 86875]. 6. Customers and social media users - Reports from customers and social media users who experienced the outage and shared their frustrations and experiences online [Article 86875].

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: - The Commonwealth Bank (CBA) experienced issues with ATMs, eftpos, and other services due to a Telstra fault [86875]. - CBA had previously faced similar issues with one of their telecommunication providers affecting some of their services [86875]. - CBA confirmed that Netbank and the CommBank app were still operational during the nationwide outage [86875]. (b) The software failure incident having happened again at multiple_organization: - ANZ bank also faced issues with its ATMs, eftpos, and other services across Australia due to the Telstra fault [86875]. - Other organizations such as Xbox, Woolworths, Australia Post, Caltex, and reportedly NSW Ambulance also experienced problems with their Telstra services [86875]. - Woolworths confirmed it was having problems processing payments but stores were still operating [86875]. - Australia Post experienced software issues in retail and urgently sought a fix from Telstra [86875].
Phase (Design/Operation) design, operation (a) The software failure incident related to the design phase can be seen in the article where a massive Telstra fault caused chaos across Australia, leading to thousands of ATMs being down and payments rejected at major retailers. The outage was believed to be caused by a severed fiber optic cable in Sydney, indicating a design vulnerability in the network infrastructure [86875]. (b) The software failure incident related to the operation phase is evident in the article where customers reported issues with ATMs, eftpos services, and other banking services across major banks like Commonwealth Bank and ANZ. This points to operational issues or misuse of the system affecting the services provided by these banks [86875].
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident was primarily within the system as it was caused by a massive Telstra fault affecting services like EFTPOS and access to ATMs for various enterprise customers nationally [86875]. The outage was believed to be due to a severed fibre optic cable in Sydney, indicating an internal infrastructure issue within the Telstra network. Additionally, various banks and organizations confirmed the outage and pointed the finger at Telstra, further emphasizing that the failure originated within the system. (b) outside_system: The software failure incident was also influenced by factors outside the system. For example, the outage impacted services like EFTPOS and access to ATMs for enterprise customers nationally, indicating a reliance on external telecommunication providers like Telstra [86875]. The article mentions that Commonwealth Bank, ANZ, NAB, and Westpac confirmed the outage and pointed the finger at Telstra, highlighting the external dependency on telecommunication providers for critical services.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident occurred due to non-human actions, specifically a massive Telstra fault causing chaos across Australia with thousands of ATMs down and payments rejected at major retailers. The outage was believed to be caused by a severed fibre optic cable in Sydney, impacting services including EFTPOS and access to ATMs for a number of enterprise customers nationally [86875]. (b) The software failure incident also involved human actions as various organizations like Commonwealth Bank, ANZ, NAB, and Westpac pointed the finger at Telstra for the outage. Commonwealth Bank, ANZ, and other banks were impacted by the issue with their telecommunication providers affecting services like ATMs and eftpos. The banks apologized to customers for the inconvenience caused by the outage and assured them that they were working towards resolving the issues [86875].
Dimension (Hardware/Software) hardware (a) The software failure incident occurred due to hardware issues, specifically a severed fibre optic cable in Sydney. Telstra, the telecommunication provider, experienced a massive fault causing chaos across Australia with thousands of ATMs down and payments rejected at major retailers [86875]. The outage affected services like EFTPOS and access to ATMs for enterprise customers nationally. Various big businesses, including Commonwealth Bank, ANZ, NAB, and Westpac, confirmed the outage and pointed the finger at Telstra [86875]. (b) The software failure incident was not directly attributed to software issues but rather to hardware problems, specifically the severed fibre optic cable in Sydney. The outage impacted services like EFTPOS and access to ATMs for enterprise customers nationally, with various big businesses confirming the outage and attributing it to Telstra [86875].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident reported in the articles does not indicate any malicious intent. The failure was attributed to a massive Telstra fault, specifically a severed fibre optic cable in Sydney, causing chaos across Australia with thousands of ATMs down and payments rejected at major retailers [86875]. The outage affected numerous big businesses with millions of customers, leading to disruptions in services such as EFTPOS and access to ATMs for enterprise customers nationally. Various organizations, including banks like Commonwealth, ANZ, NAB, and Westpac, as well as retailers like Woolworths and Australia Post, experienced issues with their services due to the Telstra fault. The incident was characterized by widespread service disruptions and inconvenience to customers, but there is no indication of deliberate harm or malicious intent behind the software failure.
Intent (Poor/Accidental Decisions) accidental_decisions (a) The software failure incident reported in the articles seems to be more related to accidental_decisions rather than poor_decisions. The incident was caused by a massive Telstra fault, specifically a severed fibre optic cable in Sydney, which led to chaos across Australia with thousands of ATMs down and payments rejected at major retailers [86875]. The outage affected numerous big businesses with millions of customers, and various organizations such as Commonwealth Bank, ANZ, NAB, Westpac, Xbox, Woolworths, Australia Post, Caltex, and NSW Ambulance experienced issues with their services due to the Telstra fault. The outage was not a result of poor decisions but rather an accidental event that impacted the telecommunications infrastructure and subsequently affected various services and businesses.
Capability (Incompetence/Accidental) accidental (a) The software failure incident reported in the articles does not specifically mention development incompetence as a contributing factor. The outage was primarily attributed to a massive Telstra fault caused by a severed fiber optic cable in Sydney, impacting services such as EFTPOS and access to ATMs for enterprise customers [86875]. (b) The software failure incident was accidental in nature, as it was caused by a severed fiber optic cable in Sydney, leading to widespread outages of ATMs and EFTPOS services across Australia for various big businesses and banks [86875]. The outage was not intentional but rather an unexpected event that affected millions of customers and businesses.
Duration temporary The software failure incident reported in Article 86875 was temporary. The incident was caused by a massive Telstra fault, specifically a severed fibre optic cable in Sydney, which led to chaos across Australia with thousands of ATMs down and payments rejected at major retailers. Telstra worked on restoring the services, and by 7pm, most services were back online, indicating a temporary nature of the failure [86875].
Behaviour crash, omission, value, other (a) crash: The software failure incident in the articles can be categorized as a crash as it resulted in ATMs going down and payments being rejected at major retailers, indicating a failure of the system to perform its intended functions [86875]. (b) omission: The software failure incident can also be categorized as an omission as the system omitted to perform its intended functions, leading to outages of ATMs and eftpos services across Australia [86875]. (c) timing: The timing of the software failure incident can be seen in the delays and disruptions caused by the system not performing its intended functions at the right time, affecting services like EFTPOS and access to ATMs for enterprise customers [86875]. (d) value: The software failure incident can be categorized as a value failure as the system performed its intended functions incorrectly, leading to issues with processing payments, withdrawals, and deposits for various organizations and customers [86875]. (e) byzantine: The software failure incident does not exhibit characteristics of a byzantine failure, which involves inconsistent responses and interactions. The incident described in the articles seems to be more of a widespread outage affecting various services due to a fault in the system [86875]. (f) other: The other behavior exhibited by the software failure incident could be described as a network failure, where the system's inability to maintain connectivity and process transactions led to disruptions in services across multiple organizations and customers [86875].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay (d) property: People's material goods, money, or data was impacted due to the software failure The software failure incident caused widespread disruptions in Australia, affecting services such as EFTPOS, ATMs, and payments at major retailers like Commonwealth Bank, ANZ, NAB, and Westpac [86875]. Customers reported issues with accessing cash, making payments, and conducting transactions due to the outage. Businesses like Woolworths had problems processing payments, and Australia Post experienced software issues affecting EFT payments, withdrawals, deposits, and balance inquiries [86875]. Additionally, customers faced difficulties in paying bills, shopping, and conducting financial transactions, highlighting the impact on people's material goods, money, and financial activities [86875].
Domain finance, other (a) The software failure incident affected the finance industry as it caused disruptions in services such as ATMs, eftpos, and other banking services provided by major banks like Commonwealth Bank, ANZ, NAB, and Westpac [86875]. Customers were unable to make transactions, leading to inconvenience and operational challenges for these financial institutions. (h) The finance industry was significantly impacted by the software failure incident, with banks like Commonwealth Bank, ANZ, NAB, and Westpac experiencing issues with their ATMs, eftpos services, and other banking functions [86875]. The outage affected millions of customers and led to disruptions in payment processing and access to financial services. (m) The software failure incident also had implications for other industries such as retail, telecommunications, and postal services. Companies like Woolworths, Xbox, Australia Post, Caltex, and NSW Ambulance reported problems with their Telstra services, affecting their operations and payment processing systems [86875]. Woolworths faced challenges in processing payments, while Australia Post experienced software issues in retail operations.

Sources

Back to List