Comment by whoamii

14 hours ago

They don’t do periodic BCDR testing?!

Which 'they?'

If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1."

Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).