Comment by yegle
13 hours ago
In Google Cloud, that's a huge no no and you'll be put on a naughty list facing serious scrutiny during the postmortem review.
Disclaimer: I work for Google Cloud.
13 hours ago
In Google Cloud, that's a huge no no and you'll be put on a naughty list facing serious scrutiny during the postmortem review.
Disclaimer: I work for Google Cloud.
Is it possible someone just tried to serve five terabytes?
No. That is definitely impossible. Nobody gets to serve five terabytes.
Ok, but why were all these higher level services that are not zonal experiencing issues then?
> In Google Cloud, that's a huge no no
When did that change? I'm glad it did, but it seems like that was long in coming.
The policy has existed for quite a while, but strict enforcement didn't come about until the great Paris outage of 2023.
And yet I had stuff in other zones that was affected.
Yes that's a good strategy if someone wants to get some face time with TK.
They don’t do periodic BCDR testing?!
Which 'they?'
If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1."
Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).
Google SRE's official (?) motto is "hope is not a strategy".
Could you explain how other regions were impacted today ?
Maybe OP is on naughty list facing serious scrutiny during the postmortem review.