| Recurring |
one_organization, multiple_organization |
(a) The software failure incident having happened again at one_organization:
- Nevada experienced a massive meltdown in the state's new computer-based test provided by test vendor Measured Progress, leading to students having trouble logging on and being repeatedly booted off the system. This incident resulted in just 30% of students completing their exams, prompting Nevada to accuse Measured Progress of breach of contract [42844].
- Nevada ultimately reached a $1.3 million settlement with Measured Progress and switched to a new vendor, Data Recognition Corp, for testing. The state superintendent mentioned that last year's problems meant that they were not able to measure student performance effectively [42844].
(b) The software failure incident having happened again at multiple_organization:
- North Dakota and Montana, also clients of Measured Progress, experienced widespread technical problems similar to Nevada's incident with the computer-based test provided by the vendor [42844].
- Minnesota, another client of Pearson, faced widespread delays and problems due to a denial-of-service attack, which was a deliberate action by hackers seeking to overload servers and slow performance. Pearson administered 50 million tests last year, with the majority completed without disruptions, but they made changes and upgrades to defend against such attacks [42844]. |
| Phase (Design/Operation) |
design, operation |
(a) The software failure incident related to the design phase can be seen in the article where it mentions technical glitches and malfunctions in computer-based testing systems used for standardized tests in various states. These glitches have disrupted testing in more than 30 states since 2013, including more than a dozen states last year and three this spring [42844]. The transition to computer-based testing aimed to move beyond simplistic multiple-choice questions but has been riddled with issues such as students not being able to log on, answers disappearing, and testing failures leading to last-minute lesson plans by teachers.
(b) The software failure incident related to the operation phase is evident in the article where it describes specific instances of testing disruptions and failures during the actual testing process. For example, Tennessee had to abandon computerized exams and return to paper-and-pencil versions due to problems on the first day of testing [42844]. Similarly, Texas faced technical hiccups that appeared to erase students' answers on more than 14,000 exams, leading the state education chief to call the testing experience "simply unacceptable." Additionally, Alaska canceled all K-12 standardized testing for the year due to chaos in schools caused by repeated testing disruptions, including a fiber-optic cable being severed by a backhoe operator in Kansas, cutting the state's connection to its test vendor [42844]. |
| Boundary (Internal/External) |
within_system, outside_system |
(a) within_system: The software failure incidents related to the transition to computer-based testing for standardized exams were primarily due to contributing factors that originated from within the system. Issues such as technical glitches, malfunctions, disruptions, erasing of students' answers, and login problems were reported across multiple states and testing companies [42844].
(b) outside_system: However, there were also instances where the software failure incidents were influenced by factors originating from outside the system. For example, in Alaska, the cancellation of standardized testing for the year was caused by a fiber-optic cable being inadvertently severed by someone operating a backhoe in Kansas, cutting the state's connection to its test vendor [42844]. |
| Nature (Human/Non-human) |
non-human_actions, human_actions |
(a) The software failure incident occurring due to non-human actions:
- In Alaska, the standardized testing was canceled for the year due to a fiber-optic cable being inadvertently severed by someone operating a backhoe in Kansas, cutting the state's connection to its test vendor [42844].
- Minnesota experienced widespread delays and problems in testing last year due to a denial-of-service attack, a deliberate action by hackers seeking to overload servers and slow their performance [42844].
(b) The software failure incident occurring due to human actions:
- Nevada officials accused test vendor Measured Progress of breach of contract after a massive meltdown in the state's new computer-based test, where students had trouble logging on and were repeatedly booted off the system [42844].
- Tennessee scrapped its computerized exams and returned to paper-and-pencil versions due to problems on the first day of testing, indicating issues with the online testing system [42844]. |
| Dimension (Hardware/Software) |
hardware, software |
(a) The software failure incident occurring due to hardware:
- In the incident reported in the articles, a significant software failure incident occurred in Alaska where standardized testing was canceled for the year due to a hardware-related issue. A backhoe accidentally severed a fiber-optic cable in Kansas, cutting Alaska's connection to its test vendor, leading to chaos in schools and repeated testing disruptions [42844].
- Additionally, in Minnesota, Pearson, the test vendor, reported that widespread delays and problems in testing were caused by a denial-of-service attack, a deliberate action by hackers seeking to overload servers and slow their performance [42844].
(b) The software failure incident occurring due to software:
- The software failure incidents reported in the articles were primarily related to issues originating in software. For example, in Nevada, a massive meltdown in the state's new computer-based test was attributed to problems with the test vendor, Measured Progress, leading to students having trouble logging on and being repeatedly booted off the system [42844].
- Furthermore, technical glitches and malfunctions in computer-based testing were reported in various states, disrupting testing and leading to concerns about the validity and reliability of the results due to dysfunctional testing administration systems [42844]. |
| Objective (Malicious/Non-malicious) |
non-malicious |
(a) The articles do not mention any software failure incidents related to malicious intent or actions by humans to harm the system. Therefore, there is no information provided about software failure incidents caused by malicious factors.
(b) The software failure incidents discussed in the articles are related to non-malicious factors such as technical glitches, technical hiccups, disruptions in testing due to Internet issues, server problems, denial-of-service attacks, and other technical challenges. These incidents have led to disruptions in testing, erasure of students' answers, inability to log on to exams, and other issues affecting the testing process [42844]. |
| Intent (Poor/Accidental Decisions) |
poor_decisions, accidental_decisions |
(a) The software failure incident related to the transition to computer-based testing for standardized exams was primarily due to poor decisions made by various stakeholders involved in the process. The decision to quickly introduce new assessments based on Common Core State Standards and other college and career-ready standards was driven by federal pressure, leading to a rushed rollout of the online testing systems [42844].
(b) Additionally, the incident involved accidental decisions or unintended consequences such as technical glitches, limited bandwidth in rural schools, aging computers with inconsistent technical support, and even electrical problems in old school buildings that affected the performance of the online testing systems [42844]. |
| Capability (Incompetence/Accidental) |
development_incompetence, accidental |
(a) The software failure incidents reported in the articles can be attributed to development incompetence. For example, in Nevada, officials accused test vendor Measured Progress of breach of contract after a massive meltdown in the state’s new computer-based test. Students had trouble logging on and were repeatedly booted off the system, leading to just 30 percent of students completing their exams [42844]. This indicates a failure on the part of the vendor in delivering a functional and reliable testing system, showcasing a lack of professional competence in the development and implementation of the software.
(b) Additionally, the incident in Alaska where all K-12 standardized testing for the year was canceled due to a fiber-optic cable being inadvertently severed by someone operating a backhoe in Kansas, cutting the state's connection to its test vendor, can be categorized as an accidental software failure incident [42844]. This accidental disruption highlights how external factors beyond the control of the development organization can lead to software failures. |
| Duration |
permanent, temporary |
The software failure incidents reported in the articles can be categorized as both permanent and temporary.
Permanent failures:
1. Tennessee scrapped its computerized exams in February and returned to the paper-and-pencil version due to problems on the first day of testing [42844].
2. Alaska officials canceled all K-12 standardized testing for the year due to chaos in schools caused by repeated testing disruptions after a fiber-optic cable was inadvertently severed [42844].
Temporary failures:
1. Nevada experienced a massive meltdown in the state's new computer-based test with students having trouble logging on and being booted off the system. Ultimately, just 30% of students completed their exams [42844].
2. Minnesota faced widespread delays and problems in testing due to a denial-of-service attack, a deliberate action by hackers seeking to overload servers [42844]. |
| Behaviour |
crash, omission, other |
(a) crash: The articles mention instances of crashes in the software failure incidents related to online testing. For example, Tennessee had to abandon computerized exams and return to paper-and-pencil versions due to problems on the first day of testing [42844]. Additionally, Texas experienced technical hiccups that appeared to erase students' answers on more than 14,000 exams, leading the state education chief to call the testing experience "simply unacceptable" [42844].
(b) omission: The software failure incidents also involved instances of omission where the system omitted to perform its intended functions. For instance, students in Nevada had trouble logging on and were repeatedly booted off the system during the computer-based test, leading to just 30% of students completing their exams [42844].
(c) timing: There are no specific instances of timing-related failures mentioned in the articles.
(d) value: The articles do not explicitly mention failures related to the system performing its intended functions incorrectly.
(e) byzantine: The software failure incidents did not exhibit behaviors of byzantine failures where the system behaves erroneously with inconsistent responses and interactions.
(f) other: The other behavior observed in the software failure incidents was the disruption of testing due to external factors such as a backhoe incident in Alaska that severed a fiber-optic cable, leading to chaos in schools and the cancellation of all K-12 standardized testing for the year [42844]. |