Incident: Verizon Wireless Data Fee Overcharging Incident.

Published Date: 2010-10-04

Postmortem Analysis
Timeline 1. The software failure incident at Verizon Wireless, where millions of customers were incorrectly charged for data they didn't use, happened over the past several years as per the article [3427]. Therefore, the estimated timeline for the software failure incident would be unknown as the exact date or month is not specified in the article.
System 1. Verizon's phone software [3427]
Responsible Organization 1. Verizon Wireless - The software failure incident was caused by a defect in Verizon's phone software, which led to incorrect data charges for millions of customers [3427].
Impacted Organization 1. Verizon Wireless customers [3427]
Software Causes 1. A defect in Verizon's phone software caused at least 15 million wireless customers to be charged data fees, even if they didn't subscribe to data plans [3427].
Non-software Causes 1. The failure incident was caused by incorrect charges due to a defect in Verizon's phone software, leading to customers being charged for data they didn't use [3427].
Impacts 1. Millions of Verizon Wireless customers were incorrectly charged data fees they didn't use, ranging from $2 to $6 per customer [3427]. 2. At least 15 million wireless customers were affected by the defect in Verizon's phone software, leading to the erroneous charges [3427]. 3. The total amount of incorrect charges could reach up to $50 million, prompting the U.S. Federal Communications Commission to investigate the issue [3427]. 4. Customers without data plans were billed for data sessions they did not initiate, leading to financial impacts on affected individuals [3427]. 5. The software failure incident resulted in Verizon issuing rebates to customers, with current customers receiving credits on upcoming bills and former customers receiving checks in the mail [3427].
Preventions 1. Implementing thorough testing procedures: Proper testing of the software could have potentially identified the defect in Verizon's phone software that caused the incorrect data charges [3427]. 2. Regular software audits: Conducting regular audits of the software code could have helped in detecting any anomalies or issues that could lead to such billing errors [3427]. 3. Improved monitoring and alert systems: Implementing robust monitoring systems that can detect unusual data charges or billing discrepancies in real-time could have helped in identifying the issue sooner and preventing it from affecting millions of customers [3427].
Fixes 1. Implement thorough testing procedures: Verizon should enhance its software testing processes to ensure that such defects in the phone software are identified before they impact customers [3427]. 2. Improve monitoring and alert systems: Setting up robust monitoring systems that can quickly detect anomalies in data usage and alert the company to investigate further can help prevent similar incidents in the future [3427]. 3. Enhance customer communication and transparency: Verizon should improve its communication with customers regarding billing and data charges to ensure transparency and prompt resolution of any billing discrepancies [3427].
References 1. Verizon Wireless statement 2. U.S. Federal Communications Commission (FCC) 3. Michele Ellison, FCC's enforcement bureau chief 4. The New York Times 5. The Cleveland Plain Dealer 6. Tom Pica, Verizon spokesman 7. FCC statement

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization (a) The software failure incident related to incorrect data charges by Verizon due to a defect in their phone software has happened within the same organization. Verizon Wireless customers were incorrectly charged data fees due to a software issue, leading to the need for rebates totaling potentially $50 million. This incident occurred over the past several years, affecting at least 15 million wireless customers [3427]. (b) The software failure incident related to incorrect data charges by Verizon does not mention similar incidents happening at other organizations or with their products and services. The focus of the article is on Verizon's issue and the actions taken to address it, including the FCC's investigation into the matter.
Phase (Design/Operation) design, operation (a) The software failure incident related to the design phase was due to a defect in Verizon's phone software that caused at least 15 million wireless customers to be incorrectly charged data fees, even if they didn't subscribe to data plans. This issue was attributed to software built into the phones causing minor data exchanges and accessing certain web links, resulting in unintended charges to customers [3427]. (b) The software failure incident related to the operation phase was seen in customers without data plans being billed for data sessions on their phones that they did not initiate. Some fees were charged when customers opened mobile web links, even though they were not receiving data and should not have incurred charges. Additionally, fees were levied without any actions from mobile phone users, indicating failures in the operation or usage of the system leading to incorrect billing [3427].
Boundary (Internal/External) within_system (a) within_system: The software failure incident at Verizon was caused by a defect in Verizon's phone software, which led to at least 15 million wireless customers being incorrectly charged data fees, even if they didn't subscribe to data plans [3427]. Verizon acknowledged that the majority of the data sessions involved minor data exchanges caused by software built into the phones, which resulted in false charges to customers [3427]. The company stated that customers who did not have data plans were billed for data sessions on their phones that they did not initiate, indicating an internal software issue within Verizon's systems [3427].
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident in this case was primarily due to non-human actions. Verizon Wireless customers were incorrectly charged data fees due to a defect in Verizon's phone software. This defect caused at least 15 million wireless customers to be charged data fees, even if they didn't subscribe to data plans. The charges were triggered by minor data exchanges caused by software built into the phones and accessing certain web links, which should not have incurred charges [3427]. (b) Human actions also played a role in this incident as Verizon Wireless had to take corrective actions to address the issue and provide rebates to affected customers. The U.S. Federal Communications Commission (FCC) investigated the matter after receiving complaints from consumers and questioned why it took Verizon two years to reimburse its customers and why greater disclosure and corrective actions did not happen sooner. The FCC's enforcement bureau chief mentioned exploring the possibility of additional penalties to ensure that companies prioritize consumer interests when billing problems occur [3427].
Dimension (Hardware/Software) software (a) The software failure incident reported in Article 3427 was primarily due to contributing factors originating in software. Verizon Wireless acknowledged that a defect in their phone software caused at least 15 million wireless customers to be incorrectly charged data fees, even if they didn't subscribe to data plans. The erroneous charges were a result of software issues within the phones, leading to customers being billed for data sessions they did not initiate or should not have incurred charges for. Verizon mentioned that most of the customers received false charges ranging from $2 to $6, which were caused by minor data exchanges and accessing certain web links due to the software built into their phones [3427].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident reported in Article 3427 was non-malicious. It was a result of a defect in Verizon's phone software that caused incorrect data charges to be applied to at least 15 million wireless customers, even if they didn't subscribe to data plans. This issue led to customers being charged for data sessions they did not initiate, including minor data exchanges caused by software built into their phones and accessing certain web links that should not have incurred charges. Verizon acknowledged the problem and stated that they have addressed the issues to prevent unintended data charges in the future [3427].
Intent (Poor/Accidental Decisions) poor_decisions (a) The software failure incident involving Verizon Wireless charging customers for data they didn't use was primarily due to poor decisions. The incident was caused by a defect in Verizon's phone software that incorrectly charged at least 15 million wireless customers for data fees, even if they didn't subscribe to data plans [3427]. This issue persisted for several years, indicating a systemic problem that stemmed from poor decisions in the software development and billing processes. The FCC also raised concerns about why it took Verizon two years to reimburse its customers and why corrective actions were not taken sooner, highlighting the consequences of poor decisions in addressing billing problems [3427].
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident related to development incompetence is evident in the case of Verizon's incorrect data charges to millions of customers. The incident was caused by a defect in Verizon's phone software, which led to at least 15 million wireless customers being charged data fees, even if they didn't subscribe to data plans [3427]. This issue persisted for several years, indicating a lack of professional competence in ensuring accurate billing and data usage tracking within the software systems. (b) The accidental aspect of the software failure incident is highlighted by the fact that customers without data plans were billed for data sessions on their phones that they did not initiate. Additionally, some fees were charged when customers opened mobile web links, even though they were not receiving data and should not have incurred charges, indicating unintentional billing errors within the software system [3427].
Duration temporary The software failure incident reported in Article 3427 regarding Verizon Wireless incorrectly charging data fees to customers can be categorized as a temporary failure. The incident was caused by a defect in Verizon's phone software, which led to at least 15 million wireless customers being charged data fees they didn't subscribe to. This issue was specific to the software glitch that caused the erroneous charges, and Verizon took corrective actions to address the problem and avoid unintended data charges in the future [3427].
Behaviour omission, value (a) crash: The software failure incident in the article is not described as a crash where the system loses state and does not perform any of its intended functions [3427]. (b) omission: The software failure incident in the article is related to omission, where the system omitted to perform its intended functions at instances. Specifically, Verizon's phone software defect caused at least 15 million wireless customers to be charged data fees, even if they didn't subscribe to data plans [3427]. (c) timing: The software failure incident in the article is not related to timing issues where the system performs its intended functions correctly but too late or too early [3427]. (d) value: The software failure incident in the article is related to value, where the system performed its intended functions incorrectly. Customers were billed for data sessions on their phones that they did not initiate, resulting in incorrect charges [3427]. (e) byzantine: The software failure incident in the article is not related to a byzantine behavior where the system behaves erroneously with inconsistent responses and interactions [3427]. (f) other: The software failure incident in the article does not fall under the other described behaviors [3427].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property (d) property: People's material goods, money, or data was impacted due to the software failure The software failure incident at Verizon Wireless resulted in millions of customers being incorrectly charged data fees they didn't use, amounting to tens of millions of dollars in total. Customers were charged for data sessions on their phones that they did not initiate, leading to false charges ranging from $2 to $6 per customer. The fees were charged over several years, and the affected customers will receive credits on their upcoming bills or checks in the mail for former customers [3427].
Domain information (a) The software failure incident reported in Article 3427 is related to the industry of information. The incident involved Verizon Wireless incorrectly charging millions of customers for data they didn't use due to a defect in the phone software, resulting in erroneous fees being applied to customers' bills [3427].

Sources

Back to List