Comment by anonymars

5 hours ago

"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously."

This very much resembles Admiral Cloudberg's write-up of the National Airport collision:

"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."

https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...

6 comments

anonymars

Reply
esafak  4 hours ago

> After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance.

What does that mean? If you knew that the precursors were why did you not set alerts?

  • cpgxiii  3 hours ago

    The "naive notion of system performance" is that the system is primarily operating in the ideal regime, and thus all deviations deserve alerts (and receive attention). The reality is that often the system is operating often enough in some failure mode that any such alerts would be regularly triggered by accepted behavior, and thus the alerts are systematically ignored/disabled/normalized.

    E.g. nominally you should never be mixing traffic types (aircraft and helicopters, civilian and military) in close proximity to a major airport and in a regime where TCAS is unlikely to offer sufficient protection. So in theory, any mixing should immediately trigger an alert and investigation to develop new procedures. But in practice, if you routinely allow such mixing under what you believe are "safe" practices (and get away with such mixing for a long time) then when a real accident happens you will have plenty of "proto-accidents" to look back on, but the warning signs from those near-accidents will have become accepted practice.