Incident: Software Glitch at Bank of New York Mellon Affects Funds

Published Date: 2015-08-31

Postmortem Analysis
Timeline 1. The software failure incident at Bank of New York Mellon (BNY Mellon) causing inaccurate values on mutual and exchange-traded funds began on August 24, as mentioned in Article 38594, which was published on August 31, 2015.
System 1. SunGard accounting system 2. Back-up system
Responsible Organization 1. Bank of New York Mellon (BNY Mellon) - The software problem at BNY Mellon caused inaccurate values on mutual and exchange-traded funds, leading to the glitch [38594].
Impacted Organization 1. Invesco PowerShares, Prudential Investments, and dozens of other mutual fund and ETF companies were impacted by the software failure incident at Bank of New York Mellon [38594].
Software Causes 1. The software failure incident at Bank of New York Mellon was caused by a software problem within the SunGard accounting system that became "corrupted" following an upgrade, leading to inaccurate values on mutual and exchange-traded funds [38594].
Non-software Causes 1. The failure incident at Bank of New York Mellon was caused by a corrupted SunGard accounting system following an upgrade, which led to inaccurate values on mutual and exchange-traded funds [38594].
Impacts 1. Inaccurate values on up to 1,200 mutual and exchange-traded funds were caused by the software problem at Bank of New York Mellon, leading to serious headaches for funds and ETF firms [38594]. 2. Investors who relied on mutual fund websites may have received outdated information, potentially making investment decisions based on stale data [38594]. 3. The glitch added to the already chaotic situation on Wall Street during a week of market volatility, impacting firms like Recon Capital Partners and causing ETF turbulence [38594]. 4. The BNY Mellon outage occurred due to a corrupted SunGard accounting system following an upgrade, leading to anger among clients and prompting firms like Recon Capital to consider alternatives such as State Street and U.S. Bancorp [38594].
Preventions 1. Regular testing and quality assurance procedures could have potentially caught the software problem at Bank of New York Mellon before it caused inaccurate values on mutual and exchange-traded funds [38594]. 2. Implementing a more robust backup system could have prevented the outage caused by the corrupted SunGard accounting system and the subsequent failure of the backup [38594].
Fixes 1. Implementing a more robust backup system to prevent failures like the one experienced by Bank of New York Mellon (BNY Mellon) after the SunGard accounting system became corrupted following an upgrade [38594]. 2. Conducting thorough testing and quality assurance procedures before implementing any software updates or changes to avoid introducing new glitches or faults [38594]. 3. Exploring alternative service providers such as State Street (STT) and U.S. Bancorp (USB) to diversify and potentially mitigate risks associated with relying solely on one provider like BNY Mellon [38594].
References 1. Bank of New York Mellon (BNY Mellon) CEO Gerald Hassell [38594] 2. Invesco PowerShares and Prudential Investments [38594] 3. Recon Capital Partners [38594] 4. Todd Rosenbluth, director of ETF and mutual fund research at S&P Capital IQ [38594] 5. BNY Mellon spokesman [38594] 6. SunGard [38594] 7. State Street (STT) and U.S. Bancorp (USB) [38594]

Software Taxonomy of Faults

Category Option Rationale
Recurring one_organization, multiple_organization (a) The software failure incident at Bank of New York Mellon (BNY Mellon) was caused by a software problem that led to inaccurate values on mutual and exchange-traded funds [38594]. The incident occurred due to a glitch in the SunGard accounting system that BNY Mellon uses, which became "corrupted" following an upgrade, and the backup system also failed. This outage angered clients and led to them considering alternatives such as State Street and U.S. Bancorp [38594]. (b) The software failure incident at BNY Mellon is an example of a technology glitch that affected multiple organizations. Invesco PowerShares, Prudential Investments, and dozens of other mutual fund and ETF companies rely on BNY Mellon data to provide their funds' value, and all of them were impacted by the inaccurate pricing information caused by the software problem at BNY Mellon [38594].
Phase (Design/Operation) design, operation (a) The software failure incident at Bank of New York Mellon (BNY Mellon) was attributed to a software problem caused by a SunGard accounting system becoming "corrupted" following an upgrade. Additionally, a back-up system also failed to prevent the issue [38594]. (b) The operation phase also played a role in the software failure incident as the outage occurred after the SunGard accounting system used by BNY Mellon became corrupted following an upgrade. This indicates that the operation or use of the system contributed to the failure [38594].
Boundary (Internal/External) within_system (a) The software failure incident at Bank of New York Mellon (BNY Mellon) was within the system. The incident was caused by a software problem within BNY Mellon's own system, specifically related to a glitch in the SunGard accounting system it uses. The issue occurred after a system upgrade, where the SunGard system became "corrupted," and the backup failed to function properly [38594].
Nature (Human/Non-human) non-human_actions, human_actions (a) The software failure incident at Bank of New York Mellon was attributed to a technology glitch caused by a software problem, which led to inaccurate values on mutual and exchange-traded funds [38594]. The glitch was not caused by the market chaos but added to the confusion during a chaotic week on Wall Street [38594]. (b) The software failure incident was exacerbated by a SunGard accounting system becoming "corrupted" following an upgrade, and a backup system also failing to work [38594]. This human action of upgrading the system contributed to the software failure incident, leading to anger among clients and prompting firms like Recon Capital to consider alternatives such as State Street and U.S. Bancorp [38594].
Dimension (Hardware/Software) software (a) The software failure incident at Bank of New York Mellon (BNY Mellon) was not directly attributed to hardware issues. The outage was caused by a software problem within BNY Mellon's systems, specifically related to a SunGard accounting system becoming "corrupted" following an upgrade, and a backup system failing as well [38594]. (b) The software failure incident at BNY Mellon was primarily due to a software problem. The glitch was caused by a software issue within the SunGard accounting system used by BNY Mellon, which led to inaccurate values on mutual and exchange-traded funds. The software problem persisted for several days, impacting various mutual fund and ETF companies that relied on BNY Mellon's data for pricing information [38594].
Objective (Malicious/Non-malicious) non-malicious (a) The software failure incident at Bank of New York Mellon (BNY Mellon) was non-malicious. The incident was caused by a software problem at BNY Mellon, specifically related to a glitch in their system that led to inaccurate values on mutual and exchange-traded funds [38594]. The outage occurred after a SunGard accounting system used by BNY Mellon became "corrupted" following an upgrade, and a backup also failed. SunGard worked to restore the system, indicating that the failure was not due to malicious intent but rather technical issues [38594].
Intent (Poor/Accidental Decisions) accidental_decisions (a) The software failure incident at Bank of New York Mellon (BNY Mellon) was not directly attributed to poor decisions but rather to a technology glitch caused by a software problem. The glitch resulted in inaccurate values on up to 1,200 mutual and exchange-traded funds, impacting various firms and investors [38594]. The incident was described as an unfortunate timing during a chaotic market situation, but there was no explicit mention of poor decisions leading to the software failure.
Capability (Incompetence/Accidental) accidental (a) The software failure incident at Bank of New York Mellon was not attributed to development incompetence but rather to a technology glitch. The glitch was caused by a software problem at the bank, leading to inaccurate values on mutual and exchange-traded funds [38594]. (b) The software failure incident at Bank of New York Mellon was accidental in nature. It was reported that the outage occurred after a SunGard accounting system used by the bank became "corrupted" following an upgrade, and a backup also failed. SunGard worked "around the clock" to restore the system, indicating that the incident was accidental rather than a result of intentional actions [38594].
Duration temporary (a) The software failure incident at Bank of New York Mellon (BNY Mellon) was temporary. The problem began on August 24 and was resolved by the following Monday [38594]. The outage caused inaccurate values on up to 1,200 mutual and exchange-traded funds, impacting firms like Invesco PowerShares and Prudential Investments. BNY Mellon was scrambling to fix the glitch throughout the week and finally provided up-to-date pricing information before markets opened on Monday. The incident was attributed to a software problem at BNY Mellon, which was caused by a corrupted SunGard accounting system following an upgrade, and a failed backup system [38594].
Behaviour crash, omission, value, other (a) crash: The software problem at Bank of New York Mellon caused inaccurate values on up to 1,200 mutual and exchange-traded funds, leading to a system crash where it was not performing its intended functions [38594]. (b) omission: The software glitch at BNY Mellon resulted in outdated pricing information being provided to investors, potentially leading to the omission of performing its intended functions correctly [38594]. (c) timing: The software failure incident occurred at a critical time when U.S. markets suffered their worst one-day plunge since 2011, indicating a timing issue where the system failed to provide accurate data when needed [38594]. (d) value: The software problem caused incorrect values on mutual and exchange-traded funds, indicating a failure in performing its intended functions correctly in terms of providing accurate values [38594]. (e) byzantine: The article does not provide specific information indicating a byzantine behavior of the software failure incident. (f) other: The software failure incident at BNY Mellon resulted in serious headaches for funds and ETF firms, impacting their operations and potentially causing confusion among investors, which could be categorized as an "other" behavior of the system failure [38594].

IoT System Layer

Layer Option Rationale
Perception None None
Communication None None
Application None None

Other Details

Category Option Rationale
Consequence property, theoretical_consequence (d) property: People's material goods, money, or data was impacted due to the software failure The software problem at Bank of New York Mellon caused inaccurate values on up to 1,200 mutual and exchange-traded funds, impacting firms, investors, and shareholders [38594]. The glitch led to outdated pricing information being displayed on mutual fund websites, potentially affecting investment decisions made based on stale data [38594]. Additionally, the outage angered clients and prompted some firms to consider alternatives such as State Street and U.S. Bancorp in the wake of the incident [38594].
Domain finance The software failure incident reported in Article 38594 is related to the finance industry [38594]. The incident occurred at Bank of New York Mellon (BNY Mellon), a financial institution that provides services related to manipulating and moving money for profit. The software glitch at BNY Mellon caused inaccurate values on mutual and exchange-traded funds, impacting various mutual fund and ETF companies that rely on BNY Mellon's data for pricing information. The outage created serious headaches for funds and ETF firms, leading to issues such as outdated information being provided to investors and the need for firms to issue press releases about pricing discrepancies. The incident also highlighted the importance of timely and accurate financial data during market volatility, emphasizing the critical role of software systems in the finance industry.

Sources

Back to List