Comment by toast0
9 hours ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
9 hours ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
I actually got to see one of those once.
For a major trans-oceanic backbone provider, at least 15ish years ago they had a mile or two between Detroit and Chicago where both ends were on the same side of the interstate highway.
But it's more frequent on DAS (Distributed Antenna Systems, AKA small-cell or micro-cell) networks.
Also the challenge of when fibers are leased (if that's still a thing, based on the networks I helped design I'd say 'probably').
They really are analogous to Lamport's "Distributed System" quip; A damaged fiber owned by a company you have never heard of can wreck your day.
In route one circuit from Washington to London via ashburn and down to Texas before going on subsea circuits through mainland Europe to ensure path diversity. The other heads north east towards Newark and and surfaces in Bude. That took two years of contractural arguments.
Fibres on the Texas link die all the time (yearly) but that’s fine.
My circuits in the far east are often carried one via dues and one via the USA.
It is possible to guarentee diversity, you just need to get specific detail and have good contractual protections.
If you design your network badly sure. Spend 3 months with BR once trying to get diverse routing out of one location to their two nearby exchanges, took a long time for them to work out what they needed to do as they had the two paths crossing in a single location for quite some time, but eventually they sorted it.
>Sometimes your multiple fiber paths end up in the same bundle
For mission critical stuff like airports I would like to think they're go for a more rigorous methodology than hope for the best on paths
lol there’s supposed to be …. But then greed gets in the way lol.
Usually it’s a combo of a tier 1 that decides that a resell agreement is “good enough” and “well just eat the loss” and then by the time stuff like this rolls around “we’ll get back to you” + a bunch of silly explanations that boil down to “you’re not gonna sue us though” start coming out lol
last time this happened in my area a local farmer was burying a cow and took out the whole towns connection
Nest door neighbor did this when he just decided on a whim to put in a new driveway and started digging with a bobcat. 10 mins in and BLAM severed the main Comcast coax serving the entire neighborhood that was running under his driveway.
Luckily we were on Century Link so weren't affected by his stupidity. lol
Years ago I was touring a POP in a small city. They were very proud to point out one fiber coming in one side of the building going South and another on the opposite side of the building going North.
Put all your backups in one basket, and then pray that nobody crushes the basket.
This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
I find intonation it’s better to select two competing provides as they are less likely to use the same ducts.
Had a few issues in some lamdlpcked counties on occasion - Kabul had all traffic routing via the same route into Pakistan for a few years, Kathmandu had an earthquake knock out both ISPs within 30 seconds of each other.
Backups via starlink global roaming work well now in particularly challenging environments.
In NL this often happens because of waterways. All starts off with good intentions, multiple providers, totally different fibers exiting the building in different directions. And then they have to cross a canal or a river. It's a 50/50 at that point whether they converge on the same conduit across the water.
That is negligence, practical if not contractual.
Over ten years ago, my employer was spinning up a new DC across the state and had three links between it and the primary DC. Two were pretty direct, but we needed a third because at one point in the 300 mile path, the two main links went within 400 meters of each other.
So how national-security-adjacent critical systems like an airport system doesn't have a larger set of backup links, and validates that they are geographically separate up until the connections, and have realtime alerting on the connection status, is surprising to me.
> validates that they are geographically separate up until the connections
This isn't relevant in this case. The problems occurred at different places. The backup fiber was, separately from the issues with the primary system, cut by a construction crew. It's not two fiber lines both cut at the same spot.
Yes the probe here is a lack of monitoring.
You can get two separate failures in short succession, where the first one hasn’t been fixed. For critical devices that’s why you have tertiary backups, ideally on different technology compel you. For one major event I had diverse fibre and satelite and microwave. Lost microwave and satelite to electronic warfare but the fibres were ok. Ofcom were really pissed about it, but nothing they could do.
2 replies →
That is fascinating. Out of curiosity, how far apart would the paths have had to remain in order to be ok with just two links? Like, I'm thinking a plane crash between those two links in the 400m zone could destory both links so a third link makes sense, but if it were 800m that would probably be fine? Or what magnitude of disaster are they trying to mitigate and how much separation would that require for just two links?
For all we know the contract says the FAA is responsible to notify Verizon if the line becomes dead. We don't know what the agreement is.