Published Date: 2018-07-26
| Postmortem Analysis | |
|---|---|
| Timeline | 1. The software failure incident reported in Article 80747 happened in January 2019. 2. The software failure incident reported in Article 73299 happened in July 2018. 3. The software failure incident reported in Article 93146 happened in December 2019. 4. The software failure incident reported in Article 79917 happened in January 2019. |
| System | 1. Amazon's Rekognition facial technology [80747, 73299, 79917] 2. Facial-recognition systems developed by Amazon [93146] |
| Responsible Organization | 1. Amazon's facial recognition technology, specifically the Rekognition software, was responsible for causing the software failure incident reported in the articles [80747, 73299, 73376, 79917]. |
| Impacted Organization | 1. Members of Congress, including Representative John Lewis and Representative Bobby L. Rush, were impacted by the software failure incident involving facial recognition technology misidentifying them as people who had been arrested [Article 73299]. 2. People of color, particularly African American women, were disproportionately misidentified in a mugshot database, raising concerns about racial bias and potential abuse by law enforcement [Article 73376]. 3. Women, the elderly, and children were more likely to be misidentified by facial recognition technology, as highlighted in a federal study on the accuracy of facial recognition systems [Article 93146]. 4. Individuals, especially darker-skinned women, were impacted by Amazon's Rekognition software inaccurately identifying gender, leading to concerns about biased results and potential harm [Article 79917]. |
| Software Causes | 1. The facial recognition software developed by Amazon, known as Rekognition, struggled with accuracy in identifying gender, especially misidentifying darker-skinned women as men [80747, 73299, 79917]. 2. The software had a higher error rate in identifying gender for darker-skinned women compared to lighter-skinned individuals, indicating bias in the algorithm [79917]. 3. The software from Amazon, as well as other companies like Microsoft, showed errors in gender classification, particularly for darker-skinned individuals, due to the training data skewing heavily towards white men [79917]. 4. The facial recognition technology, including Amazon's Rekognition, had issues with misidentifying people of color, leading to concerns about racial bias and potential abuse by law enforcement [73299, 73376, 93146]. 5. The software's accuracy varied widely based on demographics such as age, gender, and race, with African American women being more frequently misidentified in searches used by police investigators [93146]. 6. The facial-recognition algorithms exhibited demographic differentials that worsened their accuracy based on a person's age, gender, or race, as highlighted in a federal study by the National Institute of Standards and Technology [93146]. |
| Non-software Causes | 1. Lack of diversity in the training data used for the facial recognition software, skewing heavily towards white men, leading to biased results [79917, 93146]. 2. Concerns about racial bias and discrimination in the facial recognition technology, particularly in misidentifying people of color [73299, 79917, 93146]. 3. Inaccuracies in gender classification, especially for darker-skinned women, highlighting potential gender bias in the technology [79917, 93146]. 4. Ethical concerns regarding the potential misuse of facial recognition technology for mass surveillance and law enforcement purposes [73299, 79917, 93146]. |
| Impacts | 1. The software failure incident involving Amazon's facial recognition technology led to misidentifying women as men and darker-skinned women as men at significant error rates, impacting the accuracy and reliability of the technology [80747, 73299, 79917]. 2. The incident raised concerns about bias and inaccuracies in facial recognition technology, particularly in identifying individuals of different genders and skin tones, which could have serious consequences if used in law enforcement or surveillance [73299, 79917]. 3. The incident highlighted the potential for discriminatory surveillance and policing targeting communities of color, immigrants, and activists, leading to calls for halting the use of such technology in high-stakes contexts like policing and government surveillance [73299, 79917]. 4. The software failure incident underscored the need for companies like Amazon to address biases in their algorithms, improve accuracy across different demographics, and submit their models for public benchmark tests to ensure fairness and precision in facial recognition technology [79917]. 5. The incident contributed to a growing debate on regulating facial recognition technology, with concerns raised by civil rights groups, privacy advocates, lawmakers, and industry leaders about the lack of oversight, potential for abuse, and the need for national laws governing its use and privacy [79917]. |
| Preventions | 1. Conducting more comprehensive and diverse testing of the facial recognition software to identify biases and inaccuracies, as suggested by the American Civil Liberties Union (ACLU) study [Article 73299]. 2. Implementing stricter regulations and standards for facial recognition algorithms to ensure accuracy and fairness, as recommended by lawmakers and privacy advocates [Article 93146]. 3. Increasing transparency and disclosure of potential biases in the software to clients and halting its use in high-stakes contexts like policing and government surveillance, as advised by researchers [Article 79917]. |
| Fixes | 1. Conducting more comprehensive and diverse testing of the facial recognition software to ensure accuracy across different demographics and skin tones [Article 79917, Article 80747]. 2. Implementing stricter regulations and standards for facial recognition technology to address biases and inaccuracies [Article 93146]. 3. Companies like Amazon submitting their facial recognition models for public benchmark tests to ensure transparency and accountability [Article 79917]. 4. Halting the use of facial recognition technology in high-stakes contexts like policing and government surveillance until biases are addressed [Article 79917]. 5. Increasing diversity in the teams developing and testing facial recognition technology to better account for potential biases [Article 73299]. | References | 1. MIT study [80747] 2. American Civil Liberties Union (ACLU) [73299, 73376] 3. National Institute of Standards and Technology (NIST) [93146] 4. M.I.T. Media Lab [79917] |
| Category | Option | Rationale |
|---|---|---|
| Recurring | one_organization, multiple_organization | (a) The software failure incident having happened again at one_organization: - Amazon's facial recognition technology, Rekognition, faced accuracy issues in identifying gender, especially with darker-skinned women. The software incorrectly identified women as men and had higher error rates for darker-skinned women compared to competing technologies from Microsoft and IBM [80747]. - The study conducted by MIT found that Amazon's Rekognition system performed more accurately when assessing lighter-skinned faces but struggled with darker-skinned women, misidentifying their gender in about 30% of the tests [79917]. (b) The software failure incident having happened again at multiple_organization: - The American Civil Liberties Union (ACLU) reported that facial recognition technology, including Amazon's Rekognition, incorrectly matched 28 members of Congress with people who had been arrested, showing a 5% error rate among legislators. The errors disproportionately affected African-American and Latino members of Congress [73299]. - A federal study released by the National Institute of Standards and Technology (NIST) revealed that facial recognition systems misidentified people of color more often than white people, with Asian and African American individuals being up to 100 times more likely to be misidentified. The study highlighted the accuracy disparities across different algorithms developed by various companies [93146]. |
| Phase (Design/Operation) | design | (a) In the case of the software failure incident related to the development phase of design, the articles highlight issues with Amazon's facial recognition software, Rekognition. The software faced accuracy and bias challenges in recognizing gender and identifying individuals accurately based on skin tone. The study conducted by MIT revealed that the software had higher error rates in identifying darker-skinned women and incorrectly classified women as men. Amazon's response included mentioning that the study didn't use the latest version of Rekognition and that the software includes both facial analysis and facial recognition functionalities, emphasizing the distinction between the two [80747, 79917]. (b) Regarding the software failure incident related to the development phase of operation, the articles discuss how Amazon's Rekognition software struggled with accuracy issues, especially in gender classification and recognizing individuals based on skin tone. The software was found to have higher error rates for darker-skinned women, leading to concerns about biased results that could impact its use by law enforcement and in public venues. The ACLU's test of Amazon's Rekognition software also found misidentifications, particularly affecting people of color, raising concerns about racial bias and potential abuse by law enforcement [73299, 73376, 93146, 79917]. |
| Boundary (Internal/External) | within_system, outside_system | (a) within_system: - The software failure incident related to facial recognition technology developed by Amazon, known as Rekognition, had issues with accuracy and bias in identifying gender and matching faces [79917]. - The study conducted by M.I.T. Media Lab found that Amazon's Rekognition system had accuracy issues in predicting gender, especially misidentifying the gender of darker-skinned women in about 30% of tests [79917]. - The National Institute of Standards and Technology (NIST) evaluated facial-recognition systems and found that most algorithms exhibited "demographic differentials" affecting accuracy based on age, gender, or race [93146]. (b) outside_system: - The software failure incident was influenced by the lack of diversity in the training data used for the facial recognition algorithms, leading to biases in identifying gender and faces [79917]. - Concerns were raised about the potential misuse and abuse of facial recognition technology by law enforcement agencies, leading to issues of discrimination and surveillance [73299]. - Civil liberties groups and lawmakers expressed concerns about the racial bias and inaccuracies in facial recognition technology, highlighting the dangers of mass surveillance and misidentification of individuals [73299]. |
| Nature (Human/Non-human) | non-human_actions, human_actions | (a) The software failure incident occurring due to non-human actions: - The articles discuss how facial recognition technology, specifically Amazon's Rekognition software, had issues with accuracy and bias in identifying gender and matching faces, particularly with darker-skinned individuals [80747, 73299, 79917]. - The National Institute of Standards and Technology (NIST) study revealed that facial-recognition systems had demographic differentials that worsened accuracy based on a person's age, gender, or race, indicating issues with the algorithms themselves rather than human actions [93146]. - The study conducted by M.I.T. Media Lab highlighted how Amazon's Rekognition system struggled with accuracy, especially in predicting the gender of darker-skinned women, pointing to inherent flaws in the software [79917]. (b) The software failure incident occurring due to human actions: - Human actions played a role in the deployment and use of facial recognition technology by law enforcement agencies, despite concerns raised by civil liberties groups, members of Congress, and researchers about the potential for discrimination and bias in the technology [73299, 73376]. - Lawmakers expressed alarm over the biased and unreliable results of facial recognition systems, calling for reassessment and regulation of the technology's use by the government [93146]. - The American Civil Liberties Union (ACLU) conducted tests that showed misidentifications by Amazon's Rekognition software, leading to concerns about racial bias and potential abuse by law enforcement, indicating the impact of human decisions in deploying and relying on such technology [73376]. |
| Dimension (Hardware/Software) | software | (a) The articles do not provide information about a software failure incident occurring due to contributing factors originating in hardware. (b) Regarding a software failure incident, the articles discuss issues related to Amazon's facial recognition software, specifically Amazon's Rekognition system. The software faced accuracy and bias issues, especially in gender classification and identifying individuals with darker skin tones. The software incorrectly identified women as men and had higher error rates for darker-skinned women compared to lighter-skinned individuals [80747, 73299, 79917]. The study highlighted concerns about biased results, potential misuse in law enforcement, and the need for improvements in accuracy and fairness. |
| Objective (Malicious/Non-malicious) | non-malicious | (a) The articles discuss software failure incidents related to facial recognition technology developed by Amazon, which have raised concerns about biases and inaccuracies in the system. The incidents are non-malicious in nature, as they stem from issues such as racial bias in the algorithms and lack of accuracy in identifying gender and ethnicity. The failures are not attributed to intentional harm but rather to inherent flaws in the technology. 1. The articles highlight how Amazon's facial recognition technology, Rekognition, struggled with basic tests of accuracy, such as correctly identifying a person's gender, and performed more accurately with lighter-skinned faces, indicating biases in the system [Article 79917]. 2. The software misidentified people of color more often than white people, showing high error rates for certain ethnicities and genders, which could lead to discriminatory surveillance and policing [Article 93146]. 3. The facial technology had difficulty recognizing the gender of darker-skinned women and made more mistakes overall compared to competing technologies, raising concerns about accuracy and potential biases [Article 80747]. 4. The American Civil Liberties Union's test found that Amazon's Rekognition system incorrectly matched 28 members of Congress with people who had been arrested, with higher error rates for people of color, indicating issues with accuracy and potential biases [Article 73299]. |
| Intent (Poor/Accidental Decisions) | poor_decisions | (a) poor_decisions: The intent of the software failure incident can be attributed to poor decisions made by Amazon in the development and deployment of their facial recognition software. The software had issues with accuracy, especially in identifying gender and recognizing darker-skinned individuals, leading to concerns about bias and potential harm if misidentifications occur [80747, 73299, 79917]. (b) accidental_decisions: There is no specific information in the articles to suggest that the software failure incident was due to accidental decisions or unintended mistakes. |
| Capability (Incompetence/Accidental) | development_incompetence | (a) The software failure incident occurring due to development incompetence: - The articles highlight issues with Amazon's Rekognition software in accurately identifying gender and faces, especially of darker-skinned individuals, indicating a lack of professional competence in the development of the software [80747, 79917]. - The study by MIT found that Amazon's Rekognition software incorrectly identified women as men and darker-skinned women as men at high error rates, showcasing a lack of accuracy and competence in gender classification [80747]. - The research conducted by M.I.T. Media Lab revealed that Amazon's Rekognition system had accuracy issues in predicting gender, particularly misidentifying the gender of darker-skinned women, indicating a lack of precision and competence in the software's development [79917]. (b) The software failure incident occurring accidentally: - The articles do not specifically mention the software failure incident as being accidental. The issues with Amazon's Rekognition software seem to stem from inherent biases and inaccuracies rather than accidental factors. |
| Duration | temporary | (a) The articles do not provide information about a permanent software failure incident where the failure is due to contributing factors introduced by all circumstances. (b) The articles discuss temporary software failure incidents related to facial recognition technology developed by Amazon. The incidents involve issues with accuracy and bias in identifying gender and matching faces, especially for darker-skinned individuals. The errors and biases in the software were highlighted through various studies and tests conducted by organizations like MIT and the American Civil Liberties Union [80747, 73299, 73376, 93146, 79917]. |
| Behaviour | omission, value | (a) crash: - There is no specific mention of a system crash in the articles. (b) omission: - The Amazon Rekognition software incorrectly identified women as men, with a higher error rate for darker-skinned women [80747]. - Amazon's facial-recognition technology falsely identified 28 members of Congress as people who have been arrested for crimes, with a disproportionate misidentification of African-American and Latino members [73299]. - Facial-recognition systems misidentified people of color more often than white people, with African American women being falsely identified more often in certain searches [93146]. - Amazon's Rekognition system performed more accurately when assessing lighter-skinned faces, raising concerns about biased results [79917]. (c) timing: - There is no specific mention of timing-related failures in the articles. (d) value: - The Amazon Rekognition software made mistakes in identifying gender, especially with darker-skinned women [80747]. - Amazon's facial-recognition technology struggled to pass basic tests of accuracy, such as correctly identifying a person's gender, with concerns about biased results [79917]. (e) byzantine: - There is no specific mention of byzantine behavior in the articles. (f) other: - The articles do not describe any other specific behavior of the software failure incident. |
| Layer | Option | Rationale |
|---|---|---|
| Perception | None | None |
| Communication | None | None |
| Application | None | None |
| Category | Option | Rationale |
|---|---|---|
| Consequence | non-human, theoretical_consequence, other | (a) death: There were no reports of people losing their lives due to the software failure incident reported in the articles. (b) harm: The software failure incident did not result in physical harm to individuals as reported in the articles. (c) basic: The software failure incident did not impact people's access to food or shelter as reported in the articles. (d) property: The software failure incident did not result in the loss of material goods, money, or data as reported in the articles. (e) delay: The software failure incident did not cause people to postpone any activities as reported in the articles. (f) non-human: The software failure incident impacted the accuracy and bias of facial recognition technology developed by Amazon, affecting the identification of individuals based on their gender and skin tone [80747, 73299, 73376, 79917, 93146]. (g) no_consequence: The software failure incident did not have any real observed consequences in terms of physical harm, death, or significant disruptions as reported in the articles. (h) theoretical_consequence: The software failure incident raised concerns about the potential for biased and inaccurate facial recognition technology to be used by law enforcement agencies, leading to misidentifications and potential harm to individuals [73299, 73376, 79917, 93146]. (i) other: The software failure incident highlighted the need for regulation and oversight of facial recognition technology to prevent discrimination and abuse in law enforcement and public venues [73299, 73376, 79917, 93146]. |
| Domain | information, government | (a) The failed system was intended to support the information industry. The system in question was facial-recognition software developed by Amazon, which was marketed to local and federal law enforcement as a crime-fighting tool but struggled with accuracy issues, particularly in correctly identifying gender and showing biases towards different skin tones [Article 79917]. (l) The failed system was also related to the government industry. The facial-recognition technology developed by Amazon was being used by law enforcement agencies, and there were concerns raised by members of Congress about the accuracy and potential biases of the system, especially in misidentifying individuals based on race and gender [Article 73299, Article 93146]. (m) The system failure incident was not directly related to any other specific industry mentioned in the articles. |
Article ID: 80747
Article ID: 73299
Article ID: 73376
Article ID: 93146
Article ID: 79917