Comment by otterley
12 hours ago
> For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.
As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
> They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
Do you blame them? You not only have to pay double for the infra but also be charged for inter-region data transfers. And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
> You not only have to pay double for the infra but also be charged for inter-region data transfers.
Well, yeah. Redundancy isn't free. I don't see how this is controversial.
> And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
What do you mean, "IAM doesn't work"? IAM has always worked, even during control plane impairments. You cannot depend on control plane operations to work during incidents; you should plan in advance for failure and have your orchestration already set up so that you don't have to touch a thing when you have to weigh away from an AZ or region.
https://docs.aws.amazon.com/wellarchitected/latest/framework...