Incident: Faulty Software System Leads to $110 Million Loss for Trucking Contractor.

Published Date: 2021-09-10

Postmortem Analysis
Timeline 1. The software failure incident involving the faulty DRO software used by the U.S. Postal Service contractors, such as Postal Fleet Services, occurred in 2016 [Article 119373].
System 1. Dynamic Route Optimization (DRO) software - The DRO software used by the U.S. Postal Service failed, leading to issues such as flawed mileage and travel time calculations, bill payment processes, inaccurate recording of trip mileage, travel times, routing, sorting, and mail processing times, and equipment requirements [119373].
Responsible Organization 1. The U.S. Postal Service was responsible for causing the software failure incident mentioned in the article [119373].
Impacted Organization 1. Postal Fleet Services (PFS) - The trucking contractor was impacted by the faulty software, leading to a $110 million shortfall, job cuts, and lawsuits in multiple states [119373]. 2. More than 200 drivers - Allegedly went unpaid for nearly two weeks before being laid off due to the software issues, leading to financial difficulties for them [119373]. 3. National Star Route Mail Contractors Association - The trade group representing mail transportation vendors highlighted the challenges faced by its members due to the software issues [119373]. 4. Stingray Distributing - A small trucking company in California pursuing $2 million from the Postal Service due to the software problems [119373]. 5. Employees of Postal Fleet Services (PFS) - Over 1,200 drivers were laid off due to the termination of contracts caused by the software issues, leading to financial hardships for them [119373].
Software Causes 1. The failure incident was caused by problems with the "dynamic route optimization" (DRO) software, including flawed mileage and travel time calculations, bill payment processes, inaccurate recording of trip mileage, travel times, routing, sorting, and mail processing times [119373].
Non-software Causes 1. The failure incident was caused by the U.S. Postal Service's reliance on faulty software, specifically the proprietary mileage-tracking system, which led to underpayments and disputes with contractors [119373]. 2. The failure incident was exacerbated by the Postal Service's cancellation of contracts with the trucking contractor, leading to financial instability and job losses for the contractor [119373].
Impacts 1. The software failure incident led to a loss of $110 million for the Florida trucking contractor, Postal Fleet Services, over four years, resulting in the slashing of hundreds of jobs and opening the company up to lawsuits in three states [119373]. 2. More than 200 drivers alleged they went unpaid for nearly two weeks before being laid off, causing financial difficulties for them [119373]. 3. The software issues with the dynamic route optimization (DRO) software, including flawed mileage and travel time calculations, bill payment processes, and errors on both sides of the ledger, affected not only Postal Fleet Services but also other members of the National Star Route Mail Contractors Association [119373]. 4. The software failure disrupted cash flow for small businesses that are transportation vendors for the Postal Service, potentially causing financing issues and driver shortages [119373]. 5. The software failure incident resulted in the termination of over 300 contracts between the Postal Service and Postal Fleet Services, leading to layoffs and financial instability for the company [119373].
Preventions 1. Conducting thorough testing and validation of the DRO software before full implementation to identify and resolve issues early on [119373]. 2. Developing an automated payment process to ensure suppliers are paid consistently, accurately, and timely [119373]. 3. Addressing feedback and concerns raised by contractors and vendors regarding the software's functionality and accuracy [119373]. 4. Implementing a formal and transparent process for addressing payment disputes and issues with the software to prevent financial losses and contract terminations [119373].
Fixes 1. Identifying and resolving DRO issues before continuing its implementation as advised by the 2019 inspector general's report [119373]. 2. Developing an automated payment process to ensure suppliers are paid consistently, accurately, and timely as recommended by the inspector general [119373]. 3. Reevaluating the program's expected cost-savings and addressing the overpayments and underpayments frequently issued by the program [119373].
References 1. Postal Fleet Services' corporate counsel Paul Waters 2. Postal Service's inspector general 3. Greg Reed, executive director of the National Star Route Mail Contractors Association 4. Robert Cintron, the Postal Service's vice president of logistics 5. Attorneys involved in the cases 6. Lori VonWald, president of Stingray Distributing 7. Larry Weisberg, an attorney in the Florida driver suit 8. William Federman, an attorney for former PFS driver Joshua Pair 9. Transportation managers and logistics supervisors who spoke anonymously 10. Jesse Santana, an assistant manager at PFS 11. Postal Service contracting officials 12. David Hendel, an attorney representing PFS [Article 119373]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident has happened again at one_organization: The software failure incident related to the faulty DRO software has occurred again within the same organization, the U.S. Postal Service. The article mentions that the Postal Service has faced issues with the DRO software, including flawed mileage and travel time calculations, bill payment processes, and errors on both sides of the ledger, leading to overpayments and underpayments [119373]. (b) The software failure incident has happened again at multiple_organization: The article indicates that the software failure incident related to the faulty DRO software has also affected other organizations or contractors working with the Postal Service. For example, Stingray Distributing, a small trucking company in California, is pursuing a $2 million payment from the Postal Service due to issues with the DRO program [119373]. Additionally, the National Star Route Mail Contractors Association mentioned that the challenges faced by Postal Fleet Services with the DRO software are experienced by other members of the association as well [119373].
Phase (Design/Operation) design, operation (a) The software failure incident in the articles seems to be related to the design phase. The proprietary mileage-tracking system, known as "dynamic route optimization" (DRO) software, was cited as faulty by the Florida trucking contractor working with the U.S. Postal Service. The software was reported to have flaws in mileage and travel time calculations, bill payment processes, and errors on both sides of the ledger, leading to overpayments and unreliability [119373]. (b) The software failure incident also involves operational factors. The DRO software inaccurately recorded trip mileage, travel times, routing, sorting, and mail processing times, as well as equipment requirements. Additionally, the program did not provide a breakdown of the services for which contractors were being paid, leading to difficulties in reconciling invoicing problems and making it challenging for contractors to understand their payments [119373].
Boundary (Internal/External) within_system (a) The software failure incident reported in the articles is primarily within the system. The failure is attributed to the Postal Service's reliance on faulty software, specifically the dynamic route optimization (DRO) software, which led to issues such as flawed mileage and travel time calculations, bill payment processes, and errors in both overpayments and underpayments [119373]. The problems with the software were identified by the Postal Service's inspector general, indicating that the issues originated from within the system itself. Additionally, the article mentions that the software used for generating truck manifests inaccurately recorded various trip details, further highlighting internal software issues [119373]. (b) The software failure incident does not indicate any significant contributing factors that originate from outside the system. The focus of the incident is on the flaws and failures within the Postal Service's software systems, particularly the DRO software, and how these internal issues led to financial losses, job cuts, and legal disputes for the trucking contractors [119373]. The article does not mention any external factors playing a significant role in the software failure incident.
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident occurring due to non-human actions: The software failure incident in the article is primarily attributed to faulty software known as the "dynamic route optimization" (DRO) software used by the U.S. Postal Service. The DRO software had significant flaws, including inaccurate mileage and travel time calculations, flawed bill payment processes, and errors in recording trip mileage, travel times, routing, sorting, and mail processing times [119373]. These issues were identified by the Postal Service's inspector general, indicating that the failure was due to contributing factors introduced without human participation. (b) The software failure incident occurring due to human actions: Human actions also played a role in the software failure incident. The article mentions that the Postal Service had canceled contracts with a trucking contractor, Postal Fleet Services (PFS), leading to the termination of over 300 contracts and subsequent layoffs of more than 1,200 drivers. PFS had raised concerns about underpayments resulting from the faulty software with Postal Service contracting officials in prior years but did so informally, outside of formal processes, out of fear of hurting future contract bids. This informal communication and lack of formal claims may have contributed to the continuation of the software issues [119373].
Dimension (Hardware/Software) software (a) The articles do not provide information about the software failure incident occurring due to hardware-related contributing factors. (b) The software failure incident reported in the articles is related to software issues. The incident involves the U.S. Postal Service's reliance on faulty software, specifically the "dynamic route optimization" (DRO) software, which has led to significant problems for trucking contractors like Postal Fleet Services (PFS) [119373]. The software is blamed for various issues such as flawed mileage and travel time calculations, bill payment processes, errors in payment, and overpayments. The software was designed to cut transportation costs and make the Postal Service more responsive to mail volume fluctuations but has faced significant flaws since its debut in 2016, leading to financial losses for contractors and impacting their ability to pay drivers [119373].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident described in the articles does not appear to be malicious. There is no indication that the failure was due to contributing factors introduced by humans with the intent to harm the system. Instead, the failure seems to be non-malicious in nature. (b) The non-malicious software failure incident is related to the Postal Service's reliance on faulty software, specifically the "dynamic route optimization" (DRO) software, which led to issues such as flawed mileage and travel time calculations, bill payment processes, errors in overpayments, and underpayments [119373]. The failure of the software was not intentional but rather a result of significant flaws that emerged during the pilot program, inaccurate recording of trip details, and ineffective software patches to fix the issues.
Intent (Poor/Accidental Decisions) poor_decisions (a) The intent of the software failure incident: The software failure incident related to the Postal Service's reliance on faulty software for its transportation contracts appears to be more aligned with poor_decisions. The article mentions that the Postal Service implemented the Dynamic Route Optimization (DRO) software in 2016 with the intent to cut transportation costs and make the service more responsive to fluctuations in mail volume. However, significant flaws emerged during the pilot program, leading to inaccurate recording of trip mileage, travel times, routing, sorting, and mail processing times [Article 119373]. Despite these issues being identified, the Postal Service continued with the implementation of the software, leading to overpayments, underpayments, and other payment discrepancies. This indicates that the failure was a result of poor decisions made during the implementation and continuation of the software system.
Capability (Incompetence/Accidental) development_incompetence, accidental (a) The software failure incident in the article seems to be more related to development incompetence. The article mentions that the Postal Fleet Services (PFS) cited problems with the "dynamic route optimization" (DRO) software, including flawed mileage and travel time calculations, and bill payment processes. The software issues were well-documented by the Postal Service's inspector general, indicating that the software had inherent flaws that were causing payment disputes and subsequent lawsuits [119373]. (b) The software failure incident could also be attributed to accidental factors. The article highlights that the DRO software used by the Postal Service inaccurately recorded trip mileage, travel times, routing, sorting, and mail processing times, despite multiple attempts to fix it. The inspector general's report advised the Postal Service to identify and resolve DRO issues before continuing its implementation, indicating that the failures were not intentional but rather a result of unintended consequences and system errors [119373].
Duration permanent The software failure incident related to the Postal Service's reliance on faulty software for mileage-tracking, known as the dynamic route optimization (DRO) software, can be considered as both a temporary and permanent failure. Temporary: The temporary aspect of the failure is evident in the fact that the issues with the DRO software were identified during the pilot program and continued to persist over the years. The software inaccurately recorded trip mileage, travel times, routing, sorting, and mail processing times, leading to billing discrepancies and payment problems for contractors [119373]. Permanent: The permanent aspect of the failure is highlighted by the fact that the problems with the DRO software were ongoing and systemic, affecting multiple contractors and leading to significant financial losses for the trucking companies involved. The software flaws were well-documented by the Postal Service's inspector general, and despite attempts to fix the issues, the software continued to cause disruptions and financial hardships for the contractors [119373]. Therefore, the software failure incident related to the DRO software can be seen as both temporary in terms of ongoing issues and permanent due to the systemic nature of the problems and their lasting impact on the contractors.
Behaviour omission, value, other (a) crash: The software failure incident in the articles does not specifically mention a crash where the system loses state and does not perform any of its intended functions. (b) omission: The software failure incident can be related to omission as the DRO software inaccurately recorded trip mileage, travel times, routing, sorting, and mail processing times, leading to problems with payment calculations and route planning [119373]. (c) timing: The software failure incident does not directly relate to timing issues where the system performs its intended functions but too late or too early. (d) value: The software failure incident can be associated with value as the DRO program frequently issued overpayments as well as underpayments, causing financial discrepancies for the contractors [119373]. (e) byzantine: The software failure incident does not align with a byzantine behavior where the system behaves erroneously with inconsistent responses and interactions. (f) other: The software failure incident can be categorized under other as it involves a situation where the software's flaws led to driver disputes, lawsuits, financial losses, and ultimately the termination of contracts, impacting various aspects of the business operations [119373].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property (d) property: People's material goods, money, or data was impacted due to the software failure The software failure incident involving the U.S. Postal Service's reliance on faulty software led to significant financial impacts on a Florida trucking contractor, Postal Fleet Services (PFS). PFS contended that the faulty software caused them to be shorted $110 million over four years, resulting in the need to slash hundreds of jobs and opening them up to lawsuits in multiple states [Article 119373]. The software issues, particularly with the dynamic route optimization (DRO) software, led to driver disputes, lawsuits, and payment problems, ultimately affecting the financial stability of the contractor and its employees.
Domain transportation, government (a) The failed system was intended to support the transportation industry. The software system in question was the "dynamic route optimization" (DRO) software used by the U.S. Postal Service for its transportation operations [Article 119373]. The DRO software was designed to optimize transportation routes for contractors based on ever-shifting mail volumes at postal facilities, with the goal of reducing deadhead miles and improving efficiency in mail transportation [Article 119373]. The issues with the DRO software included flawed mileage and travel time calculations, errors in bill payment processes, and inaccuracies in recording trip mileage, travel times, routing, sorting, and mail processing times [Article 119373].

Sources

Back to List