Incident: Telco Network Outage Affects Vodafone and Optus Customers

Published Date: 2014-06-18

Postmortem Analysis
Timeline 1. The software failure incident happened in June 2014. [27921]
System The systems that failed in the software failure incident reported in Article 27921 were: 1. Vodafone network system 2. Optus network system [27921]
Responsible Organization 1. Vodafone [27921] 2. Optus [27921]
Impacted Organization 1. Vodafone [27921] 2. Optus [27921]
Software Causes 1. The software causes of the failure incident were related to a system error that caused some mobile customer services to be blocked, leading to the inability to use voice and data services for some Optus customers [27921].
Non-software Causes 1. Network outage affecting mobile services of customers [27921] 2. System error causing some mobile customer services to be blocked [27921]
Impacts 1. Many customers were left without access to calls or data from Vodafone and Optus [27921]. 2. Users expressed frustration on social media platforms like Twitter and Facebook [27921]. 3. Optus customers received error messages and were unable to use voice and data services [27921].
Preventions 1. Implementing robust testing procedures before deploying any network changes could have potentially prevented the software failure incident [27921]. 2. Conducting thorough risk assessments and contingency planning for network outages could have helped mitigate the impact of the incident [27921]. 3. Enhancing communication protocols and customer notification systems to provide timely updates and information during network disruptions could have improved customer satisfaction and perception of the incident [27921].
Fixes 1. Identifying and resolving the system error that caused the mobile customer services to be blocked [27921] 2. Progressive restoration of services for affected customers [27921] 3. Providing credits or compensations to affected customers as a goodwill gesture [27921]
References 1. Vodafone's Facebook page [27921] 2. Optus's Twitter account [27921] 3. Statements issued by Vodafone [27921] 4. Statements issued by Optus [27921] 5. Comments and complaints from customers on social media platforms [27921]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident having happened again at one_organization: - Vodafone experienced a network outage affecting customers, leading to service disruption and leaving many customers without access to calls or data [27921]. - Vodafone had previously faced network issues, as indicated by the statement, "We've just had confirmation that we're currently experiencing some network issues," suggesting a recurring problem [27921]. (b) The software failure incident having happened again at multiple_organization: - Optus also faced network issues similar to Vodafone, with some customers experiencing mobile services being blocked due to a system error, resulting in the inability to use voice and data services [27921]. - Both Vodafone and Optus encountered network outages around the same time, indicating a simultaneous occurrence of similar incidents at different organizations [27921].
Phase (Design/Operation) operation (a) The software failure incident seems to be related to the operation phase rather than the design phase. The articles mention issues affecting mobile services of customers, with Optus stating that a system error caused some mobile customer services to be blocked, resulting in customers being unable to use voice and data services [27921]. This indicates that the failure was more operational in nature, caused by a system error during the operation of the services.
Boundary (Internal/External) within_system (a) within_system: The software failure incident reported in the articles seems to be primarily within the system. Both Vodafone and Optus acknowledged issues with their networks and mentioned working on restoring services internally. Vodafone mentioned engineers working to restore the network, and Optus stated they were investigating the cause of the issue affecting their mobile services [27921]. These statements indicate that the failure originated from within the telcos' systems or networks.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident related to non-human actions: - The network outage affecting Vodafone and Optus customers was caused by a system error that blocked some mobile customer services, leading to the inability to use voice and data services [27921]. - Optus mentioned that the issue was unrelated to other carriers, indicating that the failure was not caused by external human actions but rather an internal system error [27921]. (b) The software failure incident related to human actions: - Customers expressed frustration on social media platforms like Twitter and Facebook, venting about the inability to make calls or access data, indicating the impact of the failure on users due to human actions [27921]. - Optus acknowledged the issue and apologized to customers for the inconvenience caused by the failure, suggesting that human actions might have contributed to the system error that blocked mobile services [27921].
Dimension (Hardware/Software) hardware, software (a) The software failure incident related to hardware: - The article mentions that Optus faced an issue where "a system error caused some mobile customer services to be blocked" [27921]. - This system error led to customers being unable to use voice and data services, indicating a hardware-related failure that originated in the system [27921]. (b) The software failure incident related to software: - Vodafone acknowledged a network outage and mentioned that engineers were working to restore the network within the hour, indicating a software-related issue [27921]. - Optus also mentioned that they were investigating the cause of an issue affecting some customers' mobile services, which points to a software-related failure [27921].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident reported in the articles does not indicate any malicious intent behind the network outages experienced by Vodafone and Optus. The issues were attributed to system errors and a network outage, with both telcos actively working to restore services and apologizing to customers for the inconvenience caused. There is no mention of any deliberate actions aimed at harming the systems ([27921]). (b) The software failure incident can be categorized as non-malicious, as it was caused by system errors and technical issues rather than any intentional harm or malicious activities by individuals. Both Vodafone and Optus acknowledged the problems, worked on resolving them, and offered compensations to affected customers. The focus was on restoring services and addressing the technical issues rather than dealing with any malicious attacks ([27921]).
Intent (Poor/Accidental Decisions) accidental_decisions (a) The software failure incident reported in the articles does not directly point to poor decisions as the intent behind the failure. The incident seems to be more related to technical issues and system errors rather than poor decision-making. (b) The software failure incident appears to be more aligned with accidental decisions or mistakes rather than poor decisions. The outage was attributed to a system error that caused mobile customer services to be blocked, leading to customers being unable to use voice and data services [27921]. Additionally, Optus mentioned that the issue was unrelated to other carriers, indicating that it was an accidental technical fault rather than a deliberate poor decision [27921].
Capability (Incompetence/Accidental) accidental (a) The software failure incident related to development incompetence is not explicitly mentioned in the provided article [27921]. (b) The software failure incident seems to be accidental as it was described as a system error that caused some mobile customer services to be blocked, leading to customers being unable to use voice and data services [27921]. This issue was unrelated to other carriers and affected an isolated group of customers [27921].
Duration temporary (a) The software failure incident reported in the articles was temporary. Both Vodafone and Optus experienced network outages affecting their customers' mobile services. Vodafone acknowledged the issue and mentioned that engineers were working to restore the network within the hour [27921]. Optus also confirmed the problem and stated that they were investigating the cause of the issue affecting some of their customers' mobile services [27921]. The outage was eventually resolved, with Optus mentioning that the majority of customers were back up and running by 9 p.m. on the same day the issue began [27921].
Behaviour crash, omission, value, other (a) crash: The software failure incident in the articles can be categorized as a crash. Both Vodafone and Optus experienced network outages, leaving customers without access to calls or data. The systems were not performing their intended functions, resulting in a loss of service for customers [27921]. (b) omission: The software failure incident can also be categorized as an omission. Customers were unable to make calls or access data during the outage, indicating that the systems omitted to perform their intended functions at that instance [27921]. (c) timing: The software failure incident does not seem to be related to timing issues. The systems were not performing their intended functions due to the outage, rather than performing them too late or too early [27921]. (d) value: The software failure incident can be categorized as a value issue. The systems were performing their intended functions incorrectly during the outage, leading to a loss of service for customers [27921]. (e) byzantine: The software failure incident does not exhibit characteristics of a byzantine failure. The systems were not behaving with inconsistent responses or interactions; instead, they were simply not functioning as expected during the outage [27921]. (f) other: The other behavior exhibited by the software failure incident is a loss of service for customers. Both Vodafone and Optus customers were unable to make calls or access data, leading to frustration and complaints on social media platforms [27921].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence delay, non-human, unknown (a) unknown (b) unknown (c) unknown (d) unknown (e) delay: The software failure incident led to customers being unable to make calls or access data, causing frustration and inconvenience [27921]. (f) non-human: The network outage impacted the mobile services of customers of Vodafone and Optus [27921]. (g) harm: unknown (h) unknown (i) unknown
Domain finance (a) The software failure incident reported in the news articles affected the telecommunications industry, specifically impacting Vodafone and Optus customers by causing network outages, disrupting calls and data services [27921]. Customers of both telcos expressed frustration on social media platforms due to the service disruption [27921]. (h) The software failure incident also had implications for the finance industry as it disrupted the ability of some customers to use voice and data services, leading to a system error that caused mobile customer services to be blocked [27921]. (m) The software failure incident was not directly related to any other industry mentioned in the options provided.

Sources

Back to List