Comment by packetslave
5 years ago
"Okay, we'll pull in a DSL line from a completely separate ISP for the out-of-band access." (guess what else is in that manhole/conduit?)
"Okay, we'll use LTE for out-of-band!" (oops, the backhaul for the cell tower goes under the same bridge as the real network)
True diversity is HARD (not unsolvable, just hard. especially at scale)!
heh i toured a large data center here in dallas and listened to them brag about all the redundant connectivity they had while standing next to the conduit where they all entered the building. One person, a pair of wire cutters, and 5 seconds and that whole datacenter is dark.
Although the difference here is that loosing connection and out-of-band for a single data center shouldn't be as catastrophic for Facebook, so your examples would be tolerable?
That's the trick, though: if you don't do that level of planning for all of your datacenters and POPs (and fiber huts out in the middle of nowhere), it's inevitable that the one you most need to access during an outage will be the one where your OOB got backhoe'd.
Murphy is a jerk.