Incident: Millions of Secure Websites Won't Load on Older Android Devices

Published Date: 2020-11-09

Postmortem Analysis
Timeline 1. The software failure incident of secure websites not loading on smartphones running Android 7.1 or older after September 2021 was revealed in the article published on November 9, 2020 [107269]. Therefore, the software failure incident happened after September 2021.
System 1. Android 7.1 and older operating systems [107269]
Responsible Organization 1. Let's Encrypt - Let's Encrypt, a certificate authority, was responsible for causing the software failure incident by changing its criteria, leading to old Android operating systems not trusting its root certificates [107269].
Impacted Organization 1. Android users running version 7.1 or older, potentially affecting over 800 million users [107269] 2. Websites certified by Let’s Encrypt, including Wikipedia, Open Street Map, Metro, Variety, and the New York Post, among others [107269]
Software Causes 1. The software cause of the failure incident was the change in criteria by Let’s Encrypt, leading to old Android operating systems not trusting its root certificates, resulting in certificate error messages on websites using Let’s Encrypt certificates [107269].
Non-software Causes 1. Changes in criteria by Let’s Encrypt regarding root certificates leading to compatibility issues with older Android operating systems [107269]. 2. Expiration of the DST Root X3 certificate on September 1, 2021, leading to the reliance solely on Let’s Encrypt's own root certificate, ISRG Root X1, causing compatibility problems with older software versions [107269].
Impacts 1. Millions of secure websites certified by Let’s Encrypt, including popular sites like Wikipedia, Open Street Map, Metro, Variety, and the New York Post, won't load on smartphones running Android 7.1 or older after September 2021, affecting over 800 million users of the old Android operating systems [107269]. 2. Around 220 million websites are estimated to be affected by the issue as old systems will fail to recognize them as secure [107269]. 3. Users with Android 7.1 or older can overcome the issue by performing a software update or by installing Firefox Mobile, which supports Android 5.0 and above and ships with its own list of trusted root certificates [107269].
Preventions 1. Ensuring backward compatibility with older Android operating systems, such as Android 7.1 and older, by maintaining support for root certificates on these systems [107269]. 2. Providing clear communication and guidance to users of older Android devices on how to address the issue, such as recommending the installation of browsers like Firefox Mobile that support a wider range of root certificates [107269]. 3. Implementing a more gradual transition plan for the change in root certificates to allow for smoother adoption by older systems and devices [107269].
Fixes 1. Performing a software update on smartphones with Android 7.1 or older could potentially fix the issue [107269]. 2. Installing Firefox Mobile, which supports Android 5.0 and above and ships with its own list of trusted root certificates, could also resolve the problem [107269].
References 1. Let’s Encrypt - the certificate authority involved in the software failure incident [107269] 2. Jacob Hoffman-Andrews - lead developer at Let's Encrypt [107269] 3. IdenTrust - the certificate authority that had a cross-signature agreement with Let’s Encrypt [107269] 4. MalwareBytes - a source quoted regarding root certificates [107269] 5. Forbes - a source estimating the number of affected websites [107269]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident related to Let's Encrypt not being trusted by older versions of Android has happened within the same organization. Let's Encrypt confirmed that older versions of Android, specifically prior to 7.1.1, will no longer trust certificates issued by Let's Encrypt due to the expiration of the DST Root X3 certificate [107269]. (b) The software failure incident related to Let's Encrypt not being trusted by older versions of Android has also affected multiple organizations. Around 220 million websites using Let's Encrypt certificates may face compatibility issues with older Android systems, impacting sites like Wikipedia, Open Street Map, Metro, Variety, and the New York Post [107269].
Phase (Design/Operation) design (a) The software failure incident in the articles is related to the design phase. The failure is due to changes in the criteria of the certificate authority Let’s Encrypt, which will lead to old Android operating systems not trusting its root certificates starting from September 2021. This change in criteria introduced by Let’s Encrypt is a contributing factor introduced during the system development phase that will impact the operation of the system [107269].
Boundary (Internal/External) within_system, outside_system (a) within_system: The software failure incident related to the Let's Encrypt certificate authority affecting old Android devices running version 7.1 and older is primarily due to changes in Let's Encrypt's root certificates. Let's Encrypt will rely solely on its own root certificate, ISRG Root X1, after the expiration of the DST Root X3 certificate on September 1, 2021. This change in root certificates is an internal decision made by Let's Encrypt, which is causing compatibility issues with older versions of Android that do not trust the new root certificate [107269]. (b) outside_system: The software failure incident is also influenced by external factors such as the widespread use of old Android operating systems. Around 66% of Android devices are running version 7.1.1 or above, but the remaining 34% that run Android 7.1 and older will be affected by the certificate error messages when visiting sites with Let's Encrypt certificates. This external factor of a large user base still on older Android versions contributes to the impact of the software failure incident [107269].
Nature (Human/Non-human) non-human_actions (a) The software failure incident occurring due to non-human actions: The software failure incident reported in the articles is primarily due to changes in the criteria set by Let's Encrypt regarding root certificates. Specifically, the issue arises from the expiration of the DST Root X3 certificate and the transition to relying solely on the ISRG Root X1 certificate by Let's Encrypt [107269]. (b) The software failure incident occurring due to human actions: The software failure incident does not seem to be directly attributed to human actions. Instead, it is a result of changes in root certificates and compatibility issues with older versions of Android operating systems [107269].
Dimension (Hardware/Software) software (a) The software failure incident reported in the articles is not due to hardware issues but rather due to contributing factors that originate in software. The issue arises from changes in criteria by Let's Encrypt regarding root certificates, which will lead to old Android operating systems not trusting its root certificates, causing certificate error messages on websites using Let's Encrypt certificates [107269]. This failure is rooted in the software changes made by Let's Encrypt and the compatibility issues with older versions of Android software.
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident in this case is non-malicious. The failure is due to a change in criteria by Let's Encrypt regarding root certificates, which will lead to old Android operating systems not trusting its root certificates after September 2021. This change is not intended to harm the system but rather to enhance security by relying solely on Let's Encrypt's own root certificate, ISRG Root X1, instead of the expiring DST Root X3 certificate [107269].
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident related to the Let's Encrypt certificate authority and old Android devices not trusting its root certificates can be attributed to poor decisions. The incident arose from Let's Encrypt's decision to rely solely on its own root certificate, ISRG Root X1, leading to compatibility issues with older software versions, particularly Android versions prior to 7.1.1. This decision ultimately resulted in millions of secure websites becoming inaccessible to users of older Android operating systems [107269].
Capability (Incompetence/Accidental) accidental (a) The software failure incident in the articles is not related to development incompetence. The issue with millions of secure websites not loading on smartphones running Android 7.1 or older after September 2021 is due to changes in root certificates by Let's Encrypt, affecting the trust of certificates on older Android systems [107269]. (b) The software failure incident can be categorized as accidental. The failure is a result of changes in root certificates by Let's Encrypt, leading to compatibility issues with older versions of Android that do not trust the new root certificate, ISRG Root X1. This change was not intentional to cause the failure but rather a consequence of the expiration of the DST Root X3 certificate [107269].
Duration temporary The software failure incident described in the articles is more aligned with a temporary failure rather than a permanent one. This is because the issue affecting millions of secure websites not loading on smartphones running Android 7.1 or older is due to a change in criteria by Let’s Encrypt regarding root certificates, specifically the expiration of the DST Root X3 certificate and the transition to the ISRG Root X1 certificate [107269]. This change will lead to certificate error messages on affected devices starting from September 2021. Users can potentially overcome this issue by performing a software update or using alternative browsers like Firefox Mobile, which supports older Android versions [107269].
Behaviour other (a) crash: The software failure incident described in the articles does not involve a crash where the system loses state and does not perform any of its intended functions. The issue is related to certificate errors on old Android devices running versions 7.1 and older when visiting websites with Let's Encrypt certificates [107269]. (b) omission: The software failure incident is not due to the system omitting to perform its intended functions at an instance(s). The issue is specifically related to certificate errors on old Android devices when trying to access websites with Let's Encrypt certificates [107269]. (c) timing: The software failure incident is not due to the system performing its intended functions correctly, but too late or too early. The issue is related to certificate errors on old Android devices running versions 7.1 and older when visiting websites with Let's Encrypt certificates [107269]. (d) value: The software failure incident is not due to the system performing its intended functions incorrectly. The issue is specifically related to certificate errors on old Android devices when trying to access websites with Let's Encrypt certificates [107269]. (e) byzantine: The software failure incident does not involve the system behaving erroneously with inconsistent responses and interactions. The issue is related to certificate errors on old Android devices running versions 7.1 and older when visiting websites with Let's Encrypt certificates [107269]. (f) other: The software failure incident involves the system failing to trust certificates issued by Let's Encrypt on old Android devices running versions 7.1 and older, leading to certificate errors when accessing websites with Let's Encrypt certificates. This behavior is specific to the compatibility issue between the old Android systems and the new root certificate used by Let's Encrypt [107269].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, theoretical_consequence (d) property: People's material goods, money, or data was impacted due to the software failure The software failure incident reported in the articles relates to the potential impact on users of old Android operating systems (specifically Android 7.1 and older) due to changes in root certificates by Let's Encrypt. As a result of these changes, millions of secure websites using Let's Encrypt certificates may not load properly on smartphones running Android 7.1 or older after September 2021. This issue could affect over 800 million users of the old Android operating systems, potentially leading to certificate error messages when visiting affected websites [107269]. Additionally, Forbes estimates that around 220 million websites may be affected by this compatibility issue, as old systems may fail to recognize them as secure [107269].
Domain information (a) The software failure incident reported in the articles is related to the information industry. The incident involves secure websites that won't load on smartphones running Android 7.1 or older due to changes in root certificates issued by Let’s Encrypt, affecting millions of users and websites that rely on these certificates for authentication and security [Article 107269].

Sources

Back to List