Comment by swiftcoder

1 day ago

The level of human negligence here is astounding. The system did exactly what it was supposed to (lock the train down if the track can't be confirmed clear), and then a human controller ordered the crew to override the system, and ordered them to proceed at an unsafe speed towards a similarly disabled train?

Usually when you look deeper at failures like this, it’s because there’s a morass of bad incentives from the surrounding human system that enforces this kind of behavior.

Here is a plausible alternative explanation:

They’ve probably seen false warnings over and over and overridden them enough times that they lost sense of the risk. They need to keep trains on schedule, using a buggy system, with a manager looking over their shoulder. They override the warnings over and over, nothing goes wrong, and they forget that it may actually be right. Look at the Challenger explosion, Chernobyl, or any other failure like this, and there’s usually a bureaucratic process that has run amok, and the individual is just following perverse incentives that have been socially normalized over time.

So I’ll wait until I have more information before I proclaim this as human negligence and the system doing exactly what it’s supposed to do. We don’t know how often it was doing the wrong thing and confusing the fuck out of the human.

  • "They need to keep trains on schedule, using a buggy system, with a manager looking over their shoulder."

    Also lots of potentially angry passengers who want to get to their destination in time. Will miss important appointments, or even the last train connection for the day and be stranded if the train drivers decide to wait a little bit longer to be on the save side.

  • I recently saw a doc on the 2013 Spanish crash and it was full of absurd system problems. The govt pinned it on the driver as best they could but it's clear that it was entirely preventable.

    • Jenga effectively. 3 others and Bob spent time destabilizing but Bob over here pulled the block before it collapsed. It's his fault.

  • > human negligence and the system doing exactly what it’s supposed to do

    Negligence is not limited to the single human who issued the order. it's pretty unlikely that their order could have caused this without a significant amount of other negligence that eroded the safeguards. Nevertheless, the negligence is human, not machine in nature.

  • I did read from HN someone from Spain said that the system crashed very often that they had to rely on humans frequently.

  • Trains in spain have been running at reduced speed, delayed or even cancelled for a while due to aging infrastructure, which might've actually caused another deadly accident back in winter, so this behavior does not surprise me.

  • It’s still gross negligence to place punctuality over safety, regardless of the reasons behind this behaviour.

    • (Un)Fortunately, finding someone who is "at fault" generally helps neither in investigating root causes nor in preventing similar problems in the future.

      Usually it's detrimental, hence (criminal) investigations of personal liability in such cases are often only allowed after systemic investigation has concluded.

      In court, you are allowed to withhold information that makes you look bad, which is simply lethally unhelpful when we want safer trains.

    • I don't mean to absolve people for negligence, but when margins are very small and paychecks may have become a necessity to survive on this planet, sometimes within the complex process the collective behaviour gets pretty weird.

      There are not that many people who (care to) see the big picture(s) and the consequences their semi-small actions may have in the long run.

Whose negligence, though? "The system" isn't just the LZB equipment - the train dispatcher is just another cog inside the same system, and should be expected to occasionally fail just like every mechanical part.

This isn't a game of Hot Potato where the last person touching it goes to jail for gross negligence. You have to look at it through a Swiss Cheese model: every individual safety layer can and will have holes, but a disaster can only occur when all of those holes align.

Why did the LZB system fail? Why did the train dispatcher use the wrong backup procedure? Why did a backup procedure with zero safety checks even exist? Why did the train operator accept this backup procedure? Why did the train allow high-speed driving under those unsafe conditions?

It does indeed seem like the train dispatcher made a big mistake, but that was far from the only thing that went wrong, and it would be myopic to use him as the scapegoat for a far larger systemic issue.

  • Overriding safety systems is itself a safety capability. But this wasn't a failure of the safety systems. It wasn't even a failure of the train drivers who had been over using safety system overrides. It was a management failure for not noticing that over use of overrides.

    • Also the train driver could have pushed back, in the radio recording with the dispatcher, he tells the driver to "proceed normally", the driver asks "proceed normally?" and the dispatcher confirms it. Here's were I would expect that better training or more availability of information to the driver would've made him push back on that decision.

      3 replies →

  • why does the dispatcher even hold the right to override the automatic system by themselves? this should involve multiple people, at least a signoff by a superior.

    • The automatic system will fail at some points so manual dispatch is needed, which is how this all happened

  • Dispatcher, conductor at that time could not have know that dispatcher ordered manual bypass ignorig the rest of the guardrails of a traffic resume procedure

    • Not OP but I'm imagining we'd ask the question why the "dispatcher ordered manual bypass ignoring the rest of the guardrails". They're saying it's all slices of cheese in a system that failed here.

    • It seems that one of the train conductors did know that it was not the procedure and tried to warn of the problem.

  • Indeed the system is all of it - all gear on another level of abstraction where it as a whole is just enabling processes, not the thing running it.

  • If you ignore a road sign and run someone over in your car today, nobody will make systemic excuses for you.

    Someone was driving the train who couldn't estimate the stopping distance relative to speed.

    If a car is stalled in the middle of the road, and you hit it, that's on you. 200% if a reflective warning triangle was up 100 meters back.

    • Trains take a long time to stop even at moderate commuter train speeds because they are so heavy, let alone high speeds of 300km/h+.

      At least one of the failures here is that the dispatcher authorized such a high speed, in other systems I am familiar with, generally the procedure with a signal fault is to crawl at a speed such that the train can stop in line of sight distances.

      In fact, that is the Spanish protocol, which for some reason was bypassed by dispatch even after the train driver had questioned it:

      > Under normal protocols, the driver would be expected to continue moving at a maximum of 40km/h according to their line of sight, and not at normal speed on a track designed for speeds of up to 200km/h.

    • If a stretch of road continues to have frequent crashes even though a sign is present, it is worth examining why that sign alone is not solving the issue.

      Air travel is as safe as it is because we don't stop the investigation at "pilot error" but instead ask why the pilot made the mistake, and why a single mistake was able to lead to loss of life.

      3 replies →

    • I don't think trains are the same as cars. Trains are supposed to run at high speeds under normal circumstances and the system is supposed to keep everyone safe. It clearly said they wouldn't be able to drive over 40 km/h if they had to operate by line of sight.

      Cars otoh don't have someone overseeing everything and directing traffic. They have to operate by sight.

    • I don't mean to pick, but it seems like you have near zero knowledge of how train systems work.

      You are 100% dependent on your safety and monitoring systems because almost every problem that could come up is beyond the horizon of your human senses. Stopping distances are in thousands of feet to miles depending on the train size and weight. Add any curves and that's far beyond your visibility.

      Then you add in management complaining that you're too slow and you need to do things faster and cheaper or you'll lose your job. Things turn from solution culture to blame culture.

      Then you add in false errors and warnings in a system that teach you to ignore them and an upper management that doesn't fix them.

      Then you have the day like hundreds of others where warnings are going off, management is yelling, but this time there is a stopped train around the corner and everyone dies.

      This is a systemic problem.

    • I wish this were true, but there is basically no consequences for drivers in the situation you described. The most egregious example I can think of near me was in 2022 when a kid was crossing a flashing crosswalk with his bike near Tacoma, and truck had stopped to let him cross. A woman ignored the crosswalk signal and used the adjacent lane to drive past the stopped truck and hit the kid, killing him. The woman was never criminally charged. She didn't even experience any consequences for her driver's license.

      https://komonews.com/news/local/13-year-old-boy-killed-while...

      2 replies →

    • > Someone was driving the train who couldn't estimate the stopping distance relative to speed.

      The stopping distance for a train this size and this fast is around 2km. Good luck estimating that by sight. There is a reason the normal protocol has a 40km/h max speed for traveling according to line of sight, which the driver was told wasn't needed.

    • Indeed, car crashes tend to just blame one (or more) of the drivers. Nobody excuses a foolish driver with systemic excuses.

      On a surely unrelated note, car travel is by a substantial margin the most dangerous mode of transport most of us ever use, and the most dangerous activity most of us routinely engage in.

I would argue that we should look at systems-level explanations. The human factor is so glaring we can get caught up in it. There are what, 50?, high speed trains in movement in Spain at the same time. Even GPS + Starlink + 1 monitor (is that 50k€?) could have prevented this. But it happened, and the question is what is going on with train security in Spain in general for people to (be able to) make such decisions.

  • After learning about a sufficient number of such "accidents" I'm now convinced the cause is never "operator error", it's always a system error. It's the humans that design (or oversee their design and implementation of) those systems that are at fault.

    • "The cause", singular, shows you're not getting the the swiss cheese model still. Trying to find a singular cause is a big mistake which prevents other causes from being addressed.

      Every hole in the cheese slices, including operator error, is important. And operator error is an important kind of error in particular, not because we need to prosecute somebody but because many forms of operator error can be mitigated using better system design, but you won't figure that out if you blind yourself to operator errors.

  • The current control system dates from the early 90s. They started rolling a modern system with more redundancies and automatic blocking in 2022 but it's still not fully deployed.

    • PZB is from 1934. Arguably always prioritizing deploying the most modern system on the highest profile lines, is what prevents us from properly rolling out the basic safety systems everywhere.

      5 replies →

  • Trains are still not able to assess properly where they are, even using satellites and inside train sensors. Of course there is research on that, but it's still an unsolved problem. (Current alternative is to put markers in the track, but that's also expensive at scale.)

    • > Trains are still not able to assess properly where they are, even using satellites and inside train sensors. Of course there is research on that, but it's still an unsolved problem

      Throwing up our hands about how hard a problem this is seems like a ridiculous position to stake. If we can design self-driving cars, that operate safely in far less controlled environments, we can figure out the linear offset of a large vehicle traveling along a physical rail running along a fixed pre-determined path.

      26 replies →

    • Yeah and that's fucking ridiculous and a damning indictment of the glaring incompetence of train companies.

      If I can have reasonably accurate gps readings of my phone while I'm on the train, there is zero reason for train companies to not have systems that are at least as accurate.

      58 replies →

    • Yes they are, those balises on tracks are the standard technique for both metro and mainline signaling systems at this point. Those balises, often combined with radio communication, are how fully autonomous trains have been operating for decades. (It's possible to use things other than balises, like track radio loops as in LZB or more recently beacons or LIDAR, but balises are standard and pretty affordable).

      8 replies →

    • > Trains are still not able to assess properly where they are [...]

      Isn't the ability to do so the basis of operation for e.g. the German LZB train protection system, which has been around since the 1960s?

      If you mean that train protection systems don't (exclusively) rely on self-reported positions, your statement makes a bit more sense, but even that is changing with e.g. newer ETCS levels which do rely on self-positioning more and more.

      1 reply →

    • Would you mind sharing some more of the why it is still an open problem? (I know I know little about this subject.) I understand satellite positioning isn't definitive and did not suggest it should be the /primary/ risk mitigating factor. But even as a tertiary or lower risk mitigating factor, this would have been a worthwhile investment. (Sorry for the bleak introduction of economics.)

      And some problems are just _open_ to outside looks. My assistant tells me that the room for error in trains on one dimension ('left - right') is mere 10s of centimeters. I understand that problem. But what track you are on, isn't that obvious to the human in the cabin? (Again, simplifying, not flaming. Production ready safety is _hard_.) So could a vision based solution help count tracks from left to right? (Assistant: "US PTC, on about 54,000 route-miles, uses GPS for the lead locomotive plus crew-entered track information and switch position".) So you use the dimension that matters most in crashes (forwards / backwards) from satellite, augmented with vision or human based left-right positioning. Some trains already have sensors to sense things like leaves on the track to adjust speeding and braking... So the sensor based approach is already there.

      Perhaps I am read as suggesting that the problem is easy, and that is not my intention. But for cars this was solved decades ago for routing and years ago for self-driving. I use that daily. Flying is organised with a very clear 6 sigma approach. Crashes are nearly always situations where events / risks stack. Trains (in Spain)...?

      Tl;dr - Safety is a function of (human; process; technology) and there is clearly something lacking for trains, when compared to other types of transport.

      2 replies →

They have a failsafe to use (visual) block backup signaling on that line and used it incorrectly. 200km/h is right at the border where you don’t need in cabin signaling.

oh man, the override thing is super dangerous. I use FSD in Tesla a lot and there is a lot of times I "override" with the gas pedal because I think it phantom breaks or being slow and there were at least 2 cases where there was legitimate reason to stop (person crossing in my blind spot).

This really reminds you of the Chernobyl station computer printing all the warnings and recommendations to shut the reactor down when they started playing around with it ... and Comrade Dyatlov overriding it.

Dyatlovs seem to be born everywhere.

Sorry, but ordering trains to drive manually after the system locks itself down, is the proper procedure. Of course you are still supposed to only send trains to free sections, but the error is in that step, not that they ordered a train to drive on it's own.

Also the fallback procedure is also aided by automatic systems and a sizable number of accidents in the past were caused by that information being faulty.

  • It is the proper procedure, and it does come (in a lot of countries, AFAIK) with a lot of guardrails:

    There will be a so-called "train order" allowing passage of a specific section ignoring the safety system warnings, which has strict validity limits both in time and in the area where it applies, usually denoted by specific hectometric mileposts

    When cab signaling is involved (LZB is one of these) and it malfunctions, the entire system normally changes to "driving on sight only" - which indeed limits the train to 40, 30, 20, sometimes - 15 km/h (see https://fr.wikipedia.org/wiki/Marche_%C3%A0_vue) or the line gets switched to lineside signaling - if it is available.

    I don't know how you could violate all of those rules - both from the driver's side and from the traffic control side - but it sounds absolutely horrible. Like an override of at least 2 (if not 3) very explicit safety margins.

    • (The following is speculation:) The issue is that there was still a fallback system operating (ASFA), so they likely assumed that there is still a safety system running, while actually it needed to be ignored, because due to virtual blocks of LZB there is then more than one train in a ASFA block. This means that the next signal shows you "clear", so you accelerate thinking you are back into normal operation, but there is actually a train in front of you, just not in the next block, but in your very own block, which is a situation that is not supposed to happen. So the issue is kind of different train control systems interacting disagreeing with each other. Ironically it would have been actual if there would not have been another safety system, because then they would have known, that they need to operate manually right away.

      1 reply →

    • There are audio recording from this incident where you can hear the "train orders" being given and acknowledged.

      They were the wrong orders apparently.

  • Why is that?

    I remember a long time ago I was stuck in a train in a field a considerable time because of a similar failure and conductor just wouldn't move unless green

    I was expecting the same here, conductor is the first to die so just refuse until the idiots fix their system

    • > Why is that?

      What exactly?

      A conductor always just gets a movement order. They are not able to assess whether the track is clear, that's the reason why the signalling system exists in the first place. How they get that movement order doesn't necessarily tell what the security steps were to back it.

      Of course an attentive conductor can notice and mention there something being weird ("Doesn't the other train normally arrive first?"), but that's a rare lucky occasion, nothing to be relied on. And they just need to swallow the answer ("No, you go first today...") anyway.

      > and conductor just wouldn't move unless green

      The conductor won't ever move unless they get a movement authority, which most of the time is a green signal.

      7 replies →

    • Many train systems have a trip-stop on signals that will automatically activate the trains's emergency brake if one passes a red signal.

      It cannot be bypassed, if an operator must go through a red they have to manually reset the e-brake on each car once it passes the signal.