← Back to context

Comment by wgjordan

10 hours ago

A real example, from Facebook's 2021 outage [1]:

> Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online.

There was one (later denied) report that a 'guy with an angle grinder' was involved in gaining access to the server cage.

[1] https://news.ycombinator.com/item?id=28762611