Published Date: 2021-07-28
| Postmortem Analysis | |
|---|---|
| Timeline | 1. The software failure incident happened in July 2021 [117261, 121024]. 2. The incident occurred in July 2021 based on the articles published in July and November 2021. |
| System | 1. ExamSoft software - The software used for the remote bar exam experienced issues related to computer memory on certain devices, causing screens to go blank and computers to crash during test sessions [117261, 118710]. 2. Video proctoring feature - The video monitoring feature of the exam software used during the online bar exam used more computer memory than anticipated, leading to crashes and disruptions for some test takers [118710]. |
| Responsible Organization | 1. The software vendor ExamSoft was responsible for causing the software failure incident during the remote bar exam by using more computer memory than anticipated, leading to computer crashes for some test takers [117261, 118710]. 2. The National Conference of Bar Examiners, the organization that designs the test, was also involved in addressing the software issues faced by test takers during the remote bar exam [117261]. |
| Impacted Organization | 1. Test takers taking the remote bar exam, including law school graduates like Nicholas Wallace and Beth Feldstein, were impacted by the software failure incident [117261, 117261]. 2. The State Bar of California, which had to adjust scores and offer retests to those who faced tech problems during the online July 2021 bar exam, was also impacted by the software failure incident [121024, 118710]. |
| Software Causes | 1. The software failure incident during the online bar exam was caused by computer memory issues on certain devices, leading to crashes and screens going blank [117261]. 2. The software failure was attributed to the video proctoring feature of the exam software using more memory than anticipated, causing some computers to crash mid-test [118710]. 3. The software provider, ExamSoft, mentioned that problems were tied to computer memory on certain devices, leading to crashes and disruptions during the exam [117261]. |
| Non-software Causes | 1. Video proctoring feature using more computer memory than anticipated [118710] 2. Computer crashes or time delays during the exam [121024] |
| Impacts | 1. Test takers encountered technology problems during the online bar exam, leading to computer crashes and lost exam time, affecting an unknown number of individuals [117261, 117261]. 2. Nearly a third of test takers faced tech problems during the July 2021 bar exam, resulting in an overall pass rate drop of nearly 8 percentage points [121024]. 3. About 2% of examinees taking California's remote bar exam in July experienced significant tech problems, with 31% facing at least one technical issue, leading to lost answers or time for some individuals [118710]. |
| Preventions | 1. Conducting thorough testing and quality assurance of the software before deploying it for high-stakes exams could have prevented the software failure incident [117261, 121024, 118710]. 2. Ensuring that the software vendor, in this case, ExamSoft, accurately estimates and tests the memory requirements of the software to prevent crashes during the exam [117261, 121024, 118710]. 3. Implementing a backup plan or contingency measures in case of software failures during the exam to minimize disruptions and ensure fairness for all test takers [117261, 121024, 118710]. |
| Fixes | 1. Improving the software to optimize computer memory usage to prevent crashes during the exam sessions [117261, 118710]. 2. Conducting thorough testing and quality assurance to ensure the software functions properly for all users [117261, 118710]. 3. Providing better support and guidance for test takers facing technical issues during the exam, such as clear instructions on how to troubleshoot and resume the exam [117261, 118710]. 4. Implementing a backup plan or contingency measures in case of software failures, such as allowing affected examinees to retake the exam section where the issue occurred [118710]. 5. Considering a return to in-person testing to avoid the technical challenges associated with remote exams [117261, 118710, 121024]. | References | 1. National Conference of Bar Examiners [117261] 2. ExamSoft [117261, 118710] 3. State Bar of California [121024, 118710] 4. Test takers who experienced the software issues [117261, 118710] |
| Category | Option | Rationale |
|---|---|---|
| Recurring | one_organization, multiple_organization | (a) The software failure incident having happened again at one_organization: - The National Conference of Bar Examiners faced technology problems during the remote bar exam for the third time, with issues reported during the July 2021 exam [117261]. - The National Conference acknowledged that the testing program did not work properly for everyone and was working with software vendor ExamSoft to address the problem [117261]. - The National Conference plans to return to exclusive in-person testing starting with the next administration in February 2022 after facing technology problems in the remote exams [117261]. (b) The software failure incident having happened again at multiple_organization: - Across the country, including California, where the remote bar exam was held, some examinees experienced computer crashes or screen issues during the exam, requiring them to restart their computers [118710]. - ExamSoft, the software provider for the remote testing, attributed the computer crashes to the video proctoring feature using more memory than anticipated, causing some computers to crash mid-test [118710]. - The State Bar of California and other states faced tech problems during the remote bar exams, leading to adjustments in scores and offering retakes for affected examinees [121024]. |
| Phase (Design/Operation) | design, operation | (a) The software failure incident occurring due to the design phase: - The software failure incidents reported in the articles were primarily related to issues with the remote bar exam software used during the attorney licensing exams [117261, 121024, 118710]. - Test takers encountered technology problems such as computer crashes, screens going blank, cameras shutting off, and having to reboot their computers during the exam sessions [117261]. - The software vendor ExamSoft acknowledged that the problems were tied to computer memory on certain devices, indicating a design flaw in the software [117261]. - The software's video monitoring feature was highlighted as a factor that used more computer memory than anticipated, leading to crashes during the exam [118710]. - The State Bar of California adjusted scores for test takers who faced tech problems based on item difficulty and performance, indicating a need to address issues introduced during the design phase of the software [121024]. (b) The software failure incident occurring due to the operation phase: - Some test takers experienced significant tech problems during the remote bar exam, resulting in lost answers or lost time [118710]. - Examinees had to restart their computers during the exam due to screens going black or blue, which could be attributed to operational issues or misuse of the system [118710]. - The State Bar of California allowed some examinees who lost time or answers to retake the exam section during which their computers crashed, indicating operational adjustments to accommodate those affected by the tech problems [118710]. - The overall pass rate on the California bar exam was impacted by the tech problems faced by test takers, leading to adjustments in scores and offering retests to those who failed due to the issues [121024]. |
| Boundary (Internal/External) | within_system | (a) within_system: The software failure incident during the remote bar exam was primarily due to issues within the system. Test takers experienced problems such as computer screens going blank, cameras shutting off, and having to reboot their computers during the exam, leading to lost exam time [117261]. The software vendor ExamSoft mentioned that the problems were tied to computer memory on certain devices, indicating an internal system issue [117261]. Additionally, the State Bar of California adjusted scores for test takers who encountered tech problems during the online exam and faced computer crashes or time delays [121024]. (b) outside_system: There is no specific mention in the articles of contributing factors originating from outside the system for the software failure incident during the remote bar exam. |
| Nature (Human/Non-human) | non-human_actions | (a) The software failure incident occurring due to non-human actions: - In the software failure incidents reported during the remote bar exams, the issues were attributed to the software using more computer memory than anticipated, causing some computers to crash mid-test [117261, 118710]. - The problems with the software led to computer screens going blank or blue during the test, requiring examinees to restart their computers to resume the exam [117261, 118710]. (b) The software failure incident occurring due to human actions: - The articles do not mention any specific instances where the software failure incidents were directly caused by human actions. The focus is more on the technical issues related to the software and computer memory [117261, 118710, 121024]. |
| Dimension (Hardware/Software) | hardware, software | (a) The software failure incident occurring due to hardware: - The software failure incidents reported during the remote bar exams were attributed to computer memory issues on certain devices, leading to crashes during the test sessions [117261, 118710]. - Some examinees experienced significant tech problems that resulted in lost answers or lost time, with about 2% encountering major problems on the high-stakes attorney licensing test [118710]. - Across the country, examinees faced computer issues where their screens went black or blue during the exam, requiring them to restart their computers to resume the test, indicating hardware-related problems [118710]. (b) The software failure incident occurring due to software: - The software failure incidents were also linked to the software provided by ExamSoft, with issues arising from the video proctoring feature using more memory than anticipated, causing some computers to crash mid-test [118710]. - The National Conference of Bar Examiners acknowledged that the testing program did not work properly for everyone, and they were working with software vendor ExamSoft to address the problem [117261]. - ExamSoft spokeswoman mentioned that they were working with jurisdictions to accommodate those who experienced verified issues to ensure all applicants could complete every session, indicating software-related challenges [117261]. |
| Objective (Malicious/Non-malicious) | non-malicious | (a) The software failure incidents reported in the articles were non-malicious. The failures were attributed to technical issues such as computer crashes, time delays, screens going blank, and software not working properly during online bar exams. These issues were caused by factors like computer memory problems, video monitoring features using more memory than anticipated, and technical glitches in the testing software provided by ExamSoft [117261, 121024, 118710]. |
| Intent (Poor/Accidental Decisions) | poor_decisions | (a) poor_decisions: The software failure incidents reported in the articles seem to be more related to poor decisions rather than accidental decisions. The incidents were caused by issues such as the software using more computer memory than anticipated, leading to crashes during the exam sessions [117261, 118710, 121024]. Additionally, the decision to implement video proctoring features that caused technical problems for test takers can be seen as a poor decision in the context of ensuring a smooth exam experience. |
| Capability (Incompetence/Accidental) | development_incompetence | (a) The software failure incident occurring due to development_incompetence: - The software issues during the remote bar exam were attributed to computer memory problems on certain devices, which affected some test takers [117261]. - ExamSoft, the software vendor, mentioned that the problems were tied to computer memory on certain devices, indicating a technical issue related to software development [117261]. - The State Bar of California found that the tech problems during the online bar exam were due to the video proctoring feature using more computer memory than anticipated, leading to crashes mid-test [118710]. (b) The software failure incident occurring accidentally: - The software failure incidents during the remote bar exams were not intentional but rather unexpected technical issues that impacted test takers [117261, 118710]. - ExamSoft attributed the computer crashes during the bar exam to the video proctoring feature using more memory than anticipated, suggesting an accidental cause of the software failure [118710]. |
| Duration | temporary | (a) The articles describe software failure incidents that can be categorized as temporary. These incidents were not permanent failures but rather temporary disruptions caused by specific circumstances such as computer memory issues during the online bar exams. For example, in the incident reported in Article 117261, test takers encountered technology problems during the online bar exam, with some experiencing computer crashes and having to reboot their computers to get the software to work again. Similarly, in Article 118710, it is mentioned that about 2% of law school graduates taking California’s remote bar exam in July experienced significant tech problems that resulted in lost answers or lost time, with the issue being attributed to the video proctoring portion of the software using more memory than anticipated, causing some computers to crash mid-test. These incidents were not permanent failures but rather temporary disruptions that affected the test takers during the exam sessions, requiring them to take immediate actions to address the issues and continue with the exams. |
| Behaviour | crash, other | (a) crash: The software failure incident described in the articles can be categorized as a crash. Test takers encountered technology problems during the online bar exam, with some experiencing computer crashes that led to their screens going blank and having to reboot their computers during test sessions [117261]. Additionally, some examinees taking the California bar exam faced significant tech problems that resulted in lost answers or lost time, with about 2% encountering major issues [118710]. (b) omission: There is no specific mention of the software failure incident being related to the system omitting to perform its intended functions at an instance(s) in the articles. (c) timing: The software failure incident is not related to the system performing its intended functions correctly, but too late or too early. (d) value: The software failure incident is not related to the system performing its intended functions incorrectly. (e) byzantine: The software failure incident is not related to the system behaving erroneously with inconsistent responses and interactions. (f) other: The software failure incident involved test takers experiencing various issues such as computer screens going blank, cameras shutting off, and having to restart their computers during the exam, leading to disruptions and loss of exam time [117261]. Additionally, the software provider ExamSoft attributed the problem to the video proctoring portion of the software using more memory than anticipated, causing some computers to crash mid-test [118710]. |
| Layer | Option | Rationale |
|---|---|---|
| Perception | None | None |
| Communication | None | None |
| Application | None | None |
| Category | Option | Rationale |
|---|---|---|
| Consequence | delay | (a) death: People lost their lives due to the software failure (b) harm: People were physically harmed due to the software failure (c) basic: People's access to food or shelter was impacted because of the software failure (d) property: People's material goods, money, or data was impacted due to the software failure (e) delay: People had to postpone an activity due to the software failure (f) non-human: Non-human entities were impacted due to the software failure (g) no_consequence: There were no real observed consequences of the software failure (h) theoretical_consequence: There were potential consequences discussed of the software failure that did not occur (i) other: Was there consequence(s) of the software failure not described in the (a to h) options? What is the other consequence(s)? The articles do not mention any consequences related to death, physical harm, impact on access to food or shelter, or impact on material goods, money, or data due to the software failure incidents discussed. The main consequences observed were related to delays in the bar exam process, loss of exam time, and potential impact on exam results for test takers who experienced technical issues during the online bar exams [117261, 121024, 118710]. |
| Domain | knowledge | (a) The software failure incident was related to the education industry, specifically the bar exam for attorney licensing [117261, 121024, 118710]. |
Article ID: 117261
Article ID: 121024
Article ID: 118710