← Back to context

Comment by otterley

5 hours ago

> For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)

Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.

As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.