Comment by bsuvc
16 hours ago
I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.
For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...
1. AWS has no cross-regional or cross-zonal data plane[1] dependencies. Customers might couple them, in violation of best practices, but AWS itself does not. It's strictly forbidden.
2. That article you linked to does not support your claim. That is a report of a single-zone incident impacting us-east-1 only. Five other AZs in the same region were still fully operational.
[1] These are not the same as control plane dependencies: you might not be able to make certain mutations, but everything should still work in the steady state.
In Google Cloud, that's a huge no no and you'll be put on a naughty list facing serious scrutiny during the postmortem review.
Disclaimer: I work for Google Cloud.
Is it possible someone just tried to serve five terabytes?
No. That is definitely impossible. Nobody gets to serve five terabytes.
Ok, but why were all these higher level services that are not zonal experiencing issues then?
> In Google Cloud, that's a huge no no
When did that change? I'm glad it did, but it seems like that was long in coming.
The policy has existed for quite a while, but strict enforcement didn't come about until the great Paris outage of 2023.
And yet I had stuff in other zones that was affected.
Yes that's a good strategy if someone wants to get some face time with TK.
They don’t do periodic BCDR testing?!
Which 'they?'
If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1."
Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).
1 reply →
Could you explain how other regions were impacted today ?
Maybe OP is on naughty list facing serious scrutiny during the postmortem review.
People who don't work on cloud services are usually very sloppy with their wording which I think is leading to some disconnect here. A "zone" is not really the unit of availability for most things people consider the "cloud" for hyperscalers. A zone is like 1-3 DCs and you are very vulnerable to stuff like localized power events. For that reason most services are not offered at the zonal level and if they are you get big disclaimers saying so and lower 9s of SLA.
Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.
For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
> For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.
As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
You are generally right, except that you don't seem to know the history of major AWS outages. Failures in AWS in us-east-1, which is the first AWS cloud region, often cause cascading worldwide failures in the whole of AWS - especially failures related to DNS and to DynamoDB, if I remember correctly. That is, AWS workloada running in, say, eu-west-1 (and in fact in the whole of AWS) often start failing or at least having greatly degraded performance/availability because of a failure in AWS infra in their core region.
If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.
It's because AWS runs their control plane out of us-east-1, particularly lambda. There are also some very popular services that can only be run out of us-east-1. It's not really a technological constraint, it's a design decision.
3 replies →
This is why Microsoft calls it an Availability Zone, each zone is made up of one or more data centers equipped with entirely independent power, cooling, and networking infrastructure. One zone should not knock out other zones. Zones are also often nearby each other just enough that they have very efficient fiber optic connections between them.
That said, that's how Microsoft handles it, I'm not sure how GCP compares.
It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.
GCP has some regions where all of the AZs are located in the same physical building. AFAIK they’ve stopped doing this in newer regions and have been migrating existing regions to have physically separate sites for each AZ.
AWS has always guaranteed that different AZs are physically separate sites, but that hasn’t always been the case with GCP.
>It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events
im only familiar with microsoft in this sense, but they have "geo-zone-redundant storage" which copies data across 3+ availability zones in the primary region, and also a secondary geographic region (i think at least 100 miles away if i remember, but maybe further). any incident would have to be massive to affect multiple geo-zones
They had a comparable problem in europe-west9 in 2023. Apparently they actually run into trouble if they lose a whole zone through a physical event. It's either thundering herd or simply insufficient capacity in the other two zones to take over all the load from the third zone.
Thundering heards during zonal outage is fairly common across the providers. You are substantially better off being multi-regional than multi-zonal.
Worked at a major ISP in the 90s with many critical infra customers. True diversity is a lot harder than many imagine.