Comment by i_like_apis

    the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this.

    this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them.

Sounds like it was the perfect storm.