Comment by YZF

Root Cause Analysis is an organizational ritual that makes people feel like they're doing something. It's more about politics than about engineering. Not only is the process not useful- more often than not the correct root cause isn't even correctly identified.

Basically a poorly designed system is going to fail somewhere. I like to think about it either as the part of the iceberg that's visible or my other analogy is lighting doesn't strike twice at the same place. A robust/well designed system is just very different than a brittle/poorly designed system. The brittle system will exhibit random failures and trying to chase them is counterproductive.

We do have systems that are fairly complex and reliable. Take an internal combustion engine as one example or an entire car or an airplane (well, one of the well designed ones). Then we build bigger systems on top of that, a city's transportation system, where we do get to scales where things "fail" all the time. Yes, some bus somewhere or some train somewhere doesn't get to a station at the correct time, because a bus breaks down or heavy traffic or whatnot. But still a well designed system is robust to those. The parts we build the systems out of are well understood and so the resulting reliability can also be understood.

With software specifically we tend to not design and build that well. We throw stuff together. Then we're surprised when they fail. And we make excuses by saying "complex systems". I often feel it's our lack of discipline and skill vs. other domains and not really that our systems are that much more complex.

EDIT: Some might argue that mechanical systems like engines are just fundamentally simpler than software. But look at something like a bearing. Just one bearing is an incredibly complex system. Bearing failure is a statistical event, some bearings fail sooner, some fail later. There are centuries of know-how in the bearings used in said engine. Mechanical engineers generally use well understood components/designs and build in margin to meet the requirements of the engine they're designing. In software we often re-invent the "bearing" or we'll pick some new unproven design for a "bearing" and we'll generally build without a good understanding of the performance or failure modes of our "bearings" and without a known pattern as to how to use them and when to use them. And then we're surprised when our "car" fails in weird ways. We also don't test to the same standard that a mechanical engineer would use for a new design. They would have a room full of widgets doing ten million cycles before they accept the design as part of a larger system. We "yolo" it...