Incident: Apple's Global Outage: App Store, iTunes, and Mac App Store.

Published Date: 2015-03-11

Postmortem Analysis
Timeline 1. The software failure incident happened on March 11, 2015. [34657]
System 1. App Store 2. Mac App Store 3. iTunes Store 4. iBooks Store 5. iCloud mail 6. TestFlight 7. Apple Support [34657]
Responsible Organization 1. An internal DNS error at Apple was responsible for causing the software failure incident [34657].
Impacted Organization 1. Users of the App Store, Mac App Store, and iTunes worldwide were impacted by the software failure incident [34657].
Software Causes 1. The software causes of the failure incident were technical problems caused by an internal DNS error at Apple, leading to the services being down for approximately eleven hours, affecting users globally [34657].
Non-software Causes 1. Internal DNS error caused by an internal DNS error at Apple [34657]
Impacts 1. Frustrated users worldwide were unable to download songs and apps from Apple's sites for approximately eleven hours, leading to potential millions of dollars in lost sales for Apple [34657]. 2. Users experienced issues such as seeing spinning circles indicating downloads in progress but nothing happening, error messages, and inability to access content in the digital stores [34657]. 3. Some users reported problems accessing iCloud mail and signing in [34657]. 4. The outage affected services beyond just the App Store, including iTunes Store, Mac App Store, and iBooks Store [34657]. 5. Based on 2014 iTunes revenue, Apple was estimated to be losing around $2 million per hour during the outage, potentially higher when considering other services [34657].
Preventions 1. Implementing robust DNS monitoring and alerting systems to quickly identify and address internal DNS errors before they escalate [34657]. 2. Conducting regular system audits and testing to proactively detect and resolve potential issues in the App Store, Mac App Store, and iTunes services [34657]. 3. Enhancing redundancy and failover mechanisms to ensure continuous service availability even in the event of technical errors or outages [34657].
Fixes 1. Fixing the internal DNS error at Apple that caused the technical problems [34657] 2. Restoring all services such as the App Store, iTunes Store, Mac App Store, and iBooks Store to full functionality [34657] 3. Addressing the issues with downloading songs, apps, and other digital content from Apple's sites [34657] 4. Resolving the errors and glitches that users were experiencing, including spinning circles indicating downloads in progress but not completing, error messages, and broken pages on the sites [34657] 5. Ensuring that iCloud mail access and sign-in issues are resolved for affected users [34657]
References 1. Apple's status page [34657] 2. Apple Support Twitter handle [34657] 3. London-based social media analyst Tim Grimes [34657] 4. Various frustrated users on social media platforms [34657] 5. The Next Web [34657]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident having happened again at one_organization: - The incident of services being down for Apple's App Store, Mac App Store, and iTunes due to an internal DNS error is a recurring issue for Apple [34657]. (b) The software failure incident having happened again at multiple_organization: - The article does not provide information about similar incidents happening at other organizations or with their products and services.
Phase (Design/Operation) design, operation (a) The software failure incident in the articles was primarily due to a design issue caused by an internal DNS error at Apple. This error led to the outage affecting the App Store, Mac App Store, and iTunes Store, preventing users from downloading songs and apps. Apple confirmed that the issue was global, impacting users in various regions like the UK, US, Canada, India, and others. The outage lasted for approximately eleven hours, resulting in significant financial losses for Apple in terms of lost sales revenue [34657]. (b) The software failure incident also had operational implications as users experienced frustration and errors while trying to download content from the digital stores. Users encountered issues such as spinning circles indicating downloads in progress but not completing, error messages, broken pages, and inability to access iCloud mail. The operational impact was evident from user complaints on social media platforms and the inability of users to make purchases or access services during the outage [34657].
Boundary (Internal/External) within_system (a) The software failure incident reported in the articles is primarily within_system. The failure was caused by an internal DNS error at Apple, affecting services like the App Store, iTunes Store, Mac App Store, and iBooks Store [34657]. The outage was attributed to technical problems within Apple's system, leading to users being unable to download songs and apps, resulting in lost sales for the company. Apple acknowledged the issue as originating from within their system and worked to restore the services for customers [34657].
Nature (Human/Non-human) non-human_actions (a) The software failure incident occurred due to non-human actions, specifically an internal DNS error at Apple. This error caused technical problems leading to the outage affecting the App Store, Mac App Store, and iTunes services globally [34657].
Dimension (Hardware/Software) software (a) The software failure incident reported in the articles was not attributed to hardware issues. The root cause of the failure was identified as an internal DNS error at Apple, indicating that the contributing factors originated in software [34657].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident reported in Article 34657 was non-malicious. The incident was attributed to an internal DNS error at Apple, causing technical problems that led to the outage affecting the App Store, Mac App Store, and iTunes. Apple confirmed the issue was not due to any malicious activity but rather an internal technical error [34657].
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident related to the Apple services being down for approximately eleven hours, affecting the App Store, Mac App Store, and iTunes, was primarily due to poor decisions. The incident was caused by an internal DNS error at Apple, as confirmed by the company in a statement. This internal error led to the unavailability of all services to customers globally, resulting in significant financial losses for Apple [34657].
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident reported in Article 34657 was due to an internal DNS error at Apple, which caused technical problems leading to the outage affecting the App Store, Mac App Store, and iTunes. This issue was attributed to development incompetence, as it was caused by a mistake or fault in the internal DNS configuration by Apple's team or organization [34657]. (b) Additionally, the incident could also be categorized as accidental, as the error was not intentional but rather an unintended consequence of the internal DNS error that occurred within Apple's systems [34657].
Duration temporary (a) The software failure incident in this case was temporary. The services of the App Store, Mac App Store, and iTunes were down for approximately eleven hours due to an internal DNS error at Apple. The outage was not permanent as Apple was able to resolve the technical problems and restore all services [34657].
Behaviour crash, omission, value, other (a) crash: The software failure incident in the articles can be categorized as a crash as users were unable to download songs and apps from Apple's sites for much of the day due to technical problems caused by an internal DNS error. The services were down for approximately eleven hours, leading to a loss in revenue for Apple [34657]. (b) omission: The software failure incident can also be categorized as an omission as users were unable to download content from the App Store, iTunes Store, Mac App Store, and iBooks Store. Users experienced issues such as seeing a spinning circle indicating that a song or app is being downloaded, but nothing happens, or receiving error messages when trying to download content [34657]. (c) timing: The software failure incident does not align with a timing failure as the issue was not related to the system performing its intended functions too late or too early. Instead, the primary issue was the system not performing its intended functions due to technical problems [34657]. (d) value: The software failure incident can be categorized as a value failure as users were unable to purchase content from the iTunes Store despite being able to view it. This indicates that the system was not performing its intended functions correctly in terms of allowing users to make purchases [34657]. (e) byzantine: The software failure incident does not align with a byzantine failure as there were no mentions of inconsistent responses or interactions from the system. The primary issue was the unavailability of services and the inability to download content [34657]. (f) other: The software failure incident can be categorized as an "other" behavior as well, specifically in terms of the impact on users and the financial losses incurred by Apple. Users expressed frustration over the outage, with some experiencing difficulties accessing iCloud mail and other services. Additionally, the financial impact on Apple was significant, with potential losses of millions of dollars in revenue due to the outage [34657].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, delay, theoretical_consequence (d) property: People's material goods, money, or data was impacted due to the software failure - The software failure incident caused frustration among users who were unable to download songs and apps from Apple's sites, resulting in lost sales for Apple, potentially costing the company millions of dollars [34657]. - Users experienced issues such as error messages, inability to download content, and broken pages on the App Store and Mac App Store [34657]. - The outage affected various services including the App Store, iTunes Store, Mac App Store, and iBooks Store, leading to users being unable to purchase content [34657]. - Based on the 2014 revenue figures, it was estimated that Apple could be losing around $2 million per hour during the outage, impacting the company financially [34657]. - Users expressed frustration on social media platforms about the inability to access the services, indicating a disruption in their ability to make purchases and access content [34657].
Domain sales (a) The failed system was related to the sales industry as it impacted the App Store, Mac App Store, and iTunes, preventing users from downloading songs and apps, resulting in lost sales for Apple [34657].

Sources

Back to List