| Recurring |
one_organization, multiple_organization |
(a) The software failure incident related to DNA analysis methods at the New York City's chief medical examiner's office is a case of a failure happening within the same organization. The lab developed two techniques for DNA analysis, high-sensitivity testing, and the Forensic Statistical Tool (FST), which were later questioned for their validity and accuracy. The FST software, developed in-house, was under scrutiny for its calculations and methodology, leading to doubts about the reliability of the DNA analysis results [62875].
(b) The articles also mention that flaws in DNA methods have led to temporary shutdowns of testing in public crime labs in other locations such as Austin, Texas, and Washington. This indicates that similar incidents related to DNA analysis methods have occurred at multiple organizations or locations, raising concerns about the accuracy and reliability of DNA analysis techniques in various forensic labs [62875]. |
| Phase (Design/Operation) |
design, operation |
(a) The software failure incident related to the design phase can be seen in the development and implementation of the DNA analysis methods at the New York City DNA laboratory. The lab introduced two techniques, high-sensitivity testing of trace DNA amounts and the Forensic Statistical Tool (FST), which involved software to interpret complex mixtures of DNA samples. However, experts raised concerns about the validity and accuracy of these methods, with one expert witness questioning the accuracy of the software source code behind one technique [62875].
(b) The software failure incident related to the operation phase can be observed in the misuse or misinterpretation of the DNA analysis results generated by the software at the New York City DNA laboratory. The FST software calculated likelihood ratios to determine the presence of a suspect's genetic material in complex DNA mixtures. However, critics highlighted flaws in the software's design, including dropping valuable data from calculations and not considering factors like shared DNA among individuals in a mixture. This led to doubts about the reliability of the software's calculations and the potential for incorrect results affecting criminal cases [62875]. |
| Boundary (Internal/External) |
within_system |
(a) within_system: The software failure incident in the DNA laboratory of New York City's chief medical examiner was primarily due to issues within the system. The failure stemmed from the development and implementation of two techniques, high-sensitivity testing of trace DNA amounts and the Forensic Statistical Tool (FST), which involved software developed in-house [62875]. The accuracy and validity of these techniques were called into question by scientists, leading to concerns about the reliability of the DNA analysis methods used in criminal cases [62875]. The software used to interpret complex mixtures of DNA was found to have flaws, with experts criticizing the design and testing of the FST [62875]. The software's inventors acknowledged errors in quantifying the amount of DNA in a sample and not considering factors like shared DNA among individuals in a mixture [62875]. Ultimately, the software failure within the system led to the exclusion of evidence produced by the high-sensitivity testing and the FST by a state judge [62875].
(b) outside_system: There is no explicit mention in the articles of the software failure incident being caused by contributing factors originating from outside the system. The focus of the failure analysis is primarily on internal issues related to the development, implementation, and validation of the DNA analysis software within the laboratory [62875]. |
| Nature (Human/Non-human) |
non-human_actions, human_actions |
(a) The software failure incident in this case is related to human actions. The failure occurred due to the development and implementation of two techniques by the DNA laboratory in New York City's chief medical examiner's office. These techniques, including high-sensitivity testing of trace DNA amounts and the Forensic Statistical Tool (FST), were developed by individuals within the lab, such as Dr. Theresa A. Caragine and Dr. Adele A. Mitchell. The FST software, which calculated the likelihood of a suspect's genetic material being present in a complex DNA mixture, was a key component in the disputed DNA analysis methods used in criminal cases [62875].
(b) The failure was also influenced by non-human actions, as the software itself played a critical role in the flawed DNA analysis methods. The software, particularly the Forensic Statistical Tool (FST), was designed to interpret complex mixtures of DNA and calculate the likelihood ratios, which were used as evidence in criminal cases. The software's source code and algorithms were under scrutiny, with experts questioning its accuracy and methodology. Issues such as dropping valuable data from calculations and potential errors in the software behavior were highlighted, indicating a failure in the software's functionality [62875]. |
| Dimension (Hardware/Software) |
software |
The software failure incident reported in the articles is related to software issues rather than hardware issues. The incident involved the use of software developed in-house by the DNA laboratory in New York City's chief medical examiner's office for DNA analysis [62875]. The software, specifically the Forensic Statistical Tool (FST), was under scrutiny for its accuracy and validity in interpreting complex mixtures of DNA samples [62875]. The FST software was criticized for dropping valuable data from its calculations, potentially affecting the likelihood assigned to the defendant's DNA being in the mixture [62875]. Additionally, the software's inventors acknowledged a margin of error in quantifying the amount of DNA in a sample, and the software did not consider that different people in a mixture might share DNA [62875]. The software code for the FST was reviewed by a computer scientist, who raised concerns about the software's behavior and recommended questioning its correctness pending further testing [62875]. The incident highlights issues with the software's design and functionality rather than hardware-related problems. |
| Objective (Malicious/Non-malicious) |
non-malicious |
(a) The software failure incident described in the articles is non-malicious. The failure was related to the DNA analysis methods used by the DNA laboratory in the office of New York City’s chief medical examiner. The lab developed two techniques, high-sensitivity testing of trace DNA amounts and the Forensic Statistical Tool (FST), to interpret complex mixtures of DNA samples. These techniques were called into question by scientists, defense lawyers, and experts, leading to concerns about their validity and accuracy in criminal cases [62875].
The failure was not due to malicious intent but rather stemmed from the complexity and potential flaws in the software and methods used for DNA analysis. The software and techniques were developed to handle small amounts of DNA and complex mixtures, but criticisms arose regarding the accuracy, assumptions, and potential errors in the calculations made by the software. The concerns raised by experts and defense lawyers led to the exclusion of evidence produced by these methods in court cases, indicating a non-malicious failure in the DNA analysis software [62875]. |
| Intent (Poor/Accidental Decisions) |
poor_decisions |
(a) The intent of the software failure incident:
- The failure of the DNA analysis software methods in the New York City DNA laboratory was primarily due to poor decisions made in the development and implementation of the software [62875].
- The lab introduced two techniques, high-sensitivity testing of trace DNA amounts and the Forensic Statistical Tool (FST), which were later found to have flaws and inaccuracies. The FST software, developed in-house, was used to interpret complex mixtures of DNA and calculate the likelihood of a suspect's genetic material being present in the mixture [62875].
- The software's inventors acknowledged a margin of error in quantifying the amount of DNA in a sample, and the software did not consider factors like shared DNA among individuals in a mixture, leading to inaccurate results [62875].
- The lab faced criticism for not conducting necessary validation studies for the high-sensitivity testing method and for not being transparent with its data, leading to doubts about the reliability of the software [62875].
- The software failure incident ultimately led to the exclusion of evidence produced by both high-sensitivity testing and the FST by a state judge, highlighting the serious consequences of the flawed software methods [62875]. |
| Capability (Incompetence/Accidental) |
development_incompetence |
(a) The software failure incident in the DNA laboratory of New York City's chief medical examiner was primarily due to development incompetence. The lab introduced two techniques, high-sensitivity testing of trace DNA amounts and the Forensic Statistical Tool (FST), which were developed in-house by scientists at the lab. However, experts raised concerns about the validity and accuracy of these methods. The FST software, developed by Dr. Theresa A. Caragine and Dr. Adele A. Mitchell, was under scrutiny for its flawed design and testing methods. Dr. Ranajit Chakraborty, who initially approved the FST, later expressed regret and stated that he would not vote for it today. The software code for the FST was found to drop valuable data from its calculations, leading to unpredictable effects on the likelihood assigned to a defendant's DNA being in a mixture [62875].
(b) The accidental aspect of the software failure incident is evident in the lab's decision to replace the two disputed DNA analysis methods with newer, more broadly used technology. The lab shelved the high-sensitivity testing and the FST and adopted STRmix, a commercially available and F.B.I.-endorsed software program for DNA mixtures that is used by dozens of public labs. This change in policy was implemented to align with changing F.B.I. standards and to stay on the cutting edge of new technology. The lab also raised the minimum sample size for sensitivity testing to almost twice the initial floor, indicating a shift towards more reliable and standardized practices to prevent similar failures in the future [62875]. |
| Duration |
temporary |
The software failure incident related to the DNA analysis methods at the New York City DNA laboratory can be categorized as a temporary failure. The lab shelved the two disputed methods and replaced them with newer technology earlier this year [62875]. The lab circulated a memo in September 2016, notifying clients that it would replace both high-sensitivity testing and the Forensic Statistical Tool (FST) on January 1 with newer technology [62875]. |
| Behaviour |
omission, value, other |
(a) crash: The software failure incident in the articles does not directly mention a crash where the system loses state and does not perform any of its intended functions.
(b) omission: The failure related to the software in the articles can be attributed to omission. The article discusses how the software omitted to perform its intended functions correctly, leading to flawed DNA analysis methods being used in criminal cases [62875].
(c) timing: The software failure incident in the articles does not directly relate to timing issues where the system performs its intended functions correctly but at the wrong time.
(d) value: The failure of the software in the articles can be categorized under performing its intended functions incorrectly. The software calculated DNA mixtures and likelihood ratios incorrectly, leading to flawed results in criminal cases [62875].
(e) byzantine: The software failure incident in the articles does not exhibit behavior characteristic of a byzantine failure, where the system behaves erroneously with inconsistent responses and interactions.
(f) other: The software failure incident in the articles can be described as a failure due to flawed algorithms and statistical methods used in DNA analysis, leading to inaccurate results and potential wrongful convictions [62875]. |