Comment by otterley
3 hours ago
Here's the official story from AWS:
https://aws.amazon.com/message/101925/
This, too, was contained to us-east-1:
"While the event started at 11:48 PM PDT on October 19 and ended at 2:20 PM PDT on October 20, there were three distinct periods of impact to customer applications. First, between 11:48 PM on October 19 and 2:40 AM on October 20, Amazon DynamoDB experienced increased API error rates in the N. Virginia (us-east-1) Region. Second, between 5:30 AM and 2:09 PM on October 20, Network Load Balancer (NLB) experienced increased connection errors for some load balancers in the N. Virginia (us-east-1) Region. This was caused by health check failures in the NLB fleet, which resulted in increased connection errors on some NLBs. Third, between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed and, while instance launches began to succeed from 10:37 AM, some newly launched instances experienced connectivity issues which were resolved by 1:50 PM."
At this point we’re playing semantic nittery. us-east-1 is their control plane. critical services that only run in us-east-1 were impacted badly. this has a cascading effect to services that run in other regions that rely on those services - even if those services themselves aren’t technically out, it’s a cascading effect. the whole fucking internet went down that day.
my original claim was that their control plane runs in us-east-1. this is a fact.
> my original claim was that their control plane runs in us-east-1. this is a fact.
The claim I'm testing is whether the impact spread to AWS services in other regions.
> At this point we’re playing semantic nittery.
It's not "semantic nittery." It's important as an engineer to be precise about the impact of incidents and how systems are architected, as opposed to making broadly sweeping (and false) claims that "[an entire] control plane runs in us-east-1" and "the whole fucking internet went down that day."
As someone who worked for AWS, I have insider knowledge that what you said is not true as a blanket statement. They generally take great care to isolate both control and data planes regionally, with rare exceptions where service data has to be globally consistent. Moreover, they have always recommended not to rely on control planes to respond to incidents: https://docs.aws.amazon.com/wellarchitected/latest/framework...