Comment by toast0
4 hours ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
4 hours ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
I actually got to see one of those once.
For a major trans-oceanic backbone provider, at least 15ish years ago they had a mile or two between Detroit and Chicago where both ends were on the same side of the interstate highway.
But it's more frequent on DAS (Distributed Antenna Systems, AKA small-cell or micro-cell) networks.
Also the challenge of when fibers are leased (if that's still a thing, based on the networks I helped design I'd say 'probably').
They really are analogous to Lamport's "Distributed System" quip; A damaged fiber owned by a company you have never heard of can wreck your day.
>Sometimes your multiple fiber paths end up in the same bundle
For mission critical stuff like airports I would like to think they're go for a more rigorous methodology than hope for the best on paths
lol there’s supposed to be …. But then greed gets in the way lol.
Usually it’s a combo of a tier 1 that decides that a resell agreement is “good enough” and “well just eat the loss” and then by the time stuff like this rolls around “we’ll get back to you” + a bunch of silly explanations that boil down to “you’re not gonna sue us though” start coming out lol
Years ago I was touring a POP in a small city. They were very proud to point out one fiber coming in one side of the building going South and another on the opposite side of the building going North.
last time this happened in my area a local farmer was burying a cow and took out the whole towns connection
Nest door neighbor did this when he just decided on a whim to put in a new driveway and started digging with a bobcat. 10 mins in and BLAM severed the main Comcast coax serving the entire neighborhood that was running under his driveway.
Luckily we were on Century Link so weren't affected by his stupidity. lol
Put all your backups in one basket, and then pray that nobody crushes the basket.
This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
In NL this often happens because of waterways. All starts off with good intentions, multiple providers, totally different fibers exiting the building in different directions. And then they have to cross a canal or a river. It's a 50/50 at that point whether they converge on the same conduit across the water.
That is negligence, practical if not contractual.
Over ten years ago, my employer was spinning up a new DC across the state and had three links between it and the primary DC. Two were pretty direct, but we needed a third because at one point in the 300 mile path, the two main links went within 400 meters of each other.
So how national-security-adjacent critical systems like an airport system doesn't have a larger set of backup links, and validates that they are geographically separate up until the connections, and have realtime alerting on the connection status, is surprising to me.
> validates that they are geographically separate up until the connections
This isn't relevant in this case. The problems occurred at different places. The backup fiber was, separately from the issues with the primary system, cut by a construction crew. It's not two fiber lines both cut at the same spot.
For all we know the contract says the FAA is responsible to notify Verizon if the line becomes dead. We don't know what the agreement is.