Hacker News Reader

Comment by Anon1096 | Hacker News Reader

People who don't work on cloud services are usually very sloppy with their wording which I think is leading to some disconnect here. A "zone" is not really the unit of availability for most things people consider the "cloud" for hyperscalers. A zone is like 1-3 DCs and you are very vulnerable to stuff like localized power events. For that reason most services are not offered at the zonal level and if they are you get big disclaimers saying so and lower 9s of SLA.

Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.

For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)

View on HN

You are generally right, except that you don't seem to know the history of major AWS outages. Failures in AWS in us-east-1, which is the first AWS cloud region, often cause cascading worldwide failures in the whole of AWS - especially failures related to DNS and to DynamoDB, if I remember correctly. That is, AWS workloada running in, say, eu-west-1 (and in fact in the whole of AWS) often start failing or at least having greatly degraded performance/availability because of a failure in AWS infra in their core region.

If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.

It's because AWS runs their control plane out of us-east-1, particularly lambda. There are also some very popular services that can only be run out of us-east-1. It's not really a technological constraint, it's a design decision.

AWS does not run “their control plane” out of us-east-1. Every single service has its own control plane independently hosted in its own region, with the exceptions of IAM, Route53, Cloudfront, and a small number of others like Global Accelerator.

IAM is usually the culprit when us-east-1 issues affect other regions, but Lambda runs independently in every region.

Explain why how why when aws lambda when out in us-east-1 a year or so ago every other region had intense issues they couldn’t quickly recover from? and their incident report explained this exact fact? where’s your info coming from? I follow this stuff very closely because my livelihood depends on it.

Do you have a link to this incident report? I don’t recognize this one.

https://aws.plainenglish.io/aws-outage-october-20-2025-a-cas...

here’s one third party, it was pretty well documented as a control plane failure in us-east-1

Here's the official story from AWS:

https://aws.amazon.com/message/101925/

This, too, was contained to us-east-1:

"While the event started at 11:48 PM PDT on October 19 and ended at 2:20 PM PDT on October 20, there were three distinct periods of impact to customer applications. First, between 11:48 PM on October 19 and 2:40 AM on October 20, Amazon DynamoDB experienced increased API error rates in the N. Virginia (us-east-1) Region. Second, between 5:30 AM and 2:09 PM on October 20, Network Load Balancer (NLB) experienced increased connection errors for some load balancers in the N. Virginia (us-east-1) Region. This was caused by health check failures in the NLB fleet, which resulted in increased connection errors on some NLBs. Third, between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed and, while instance launches began to succeed from 10:37 AM, some newly launched instances experienced connectivity issues which were resolved by 1:50 PM."

At this point we’re playing semantic nittery. us-east-1 is their control plane. critical services that only run in us-east-1 were impacted badly. this has a cascading effect to services that run in other regions that rely on those services - even if those services themselves aren’t technically out, it’s a cascading effect. the whole fucking internet went down that day.

my original claim was that their control plane runs in us-east-1. this is a fact.

> my original claim was that their control plane runs in us-east-1. this is a fact.

The claim I'm testing is whether the impact spread to AWS services in other regions.

> At this point we’re playing semantic nittery.

It's not "semantic nittery." It's important as an engineer to be precise about the impact of incidents and how systems are architected, as opposed to making broadly sweeping (and false) claims that "[an entire] control plane runs in us-east-1" and "the whole fucking internet went down that day."

As someone who worked for AWS, I have insider knowledge that what you said is not true as a blanket statement. They generally take great care to isolate both control and data planes regionally, with rare exceptions where service data has to be globally consistent. Moreover, they have always recommended not to rely on control planes to respond to incidents: https://docs.aws.amazon.com/wellarchitected/latest/framework...

Wait so then how are they signing big customers still who would definitely lose a lot if us-east-1 went down?

(who would be motivated to check if anything is restricted to only us-east-1)

> For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)

Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.

As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.

> They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.

Do you blame them? You not only have to pay double for the infra but also be charged for inter-region data transfers. And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.

> You not only have to pay double for the infra but also be charged for inter-region data transfers.

Well, yeah. Redundancy isn't free. I don't see how this is controversial.

> And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.

What do you mean, "IAM doesn't work"? IAM has always worked, even during control plane impairments. You cannot depend on control plane operations to work during incidents; you should plan in advance for failure and have your orchestration already set up so that you don't have to touch a thing when you have to weigh away from an AZ or region.

https://docs.aws.amazon.com/wellarchitected/latest/framework...

> What do you mean, "IAM doesn't work"? IAM has always worked

What do you mean, "IAM always worked"? The last failure was very recent, only a year ago[0]

[0] https://www.dataminded.com/resources/how-to-protect-against-...

This points to what I keep repeating about the control plane vs. the data plane. IAM faulted to a steady state. While you couldn’t make changes during the impairment, it didn’t cause downstream failures like blocking rule evaluations (or worse, failing open and allowing denied operations to be permitted). That’s the important part.

As the page you linked to stated:

> IAM (Identity and Access Management) configuration was unavailable, preventing updates to roles and policies, but cached credentials allowed most running workloads to continue.

Perhaps I should have been clearer about what I meant about IAM still working. That's on me.

This is why Microsoft calls it an Availability Zone, each zone is made up of one or more data centers equipped with entirely independent power, cooling, and networking infrastructure. One zone should not knock out other zones. Zones are also often nearby each other just enough that they have very efficient fiber optic connections between them.

That said, that's how Microsoft handles it, I'm not sure how GCP compares.

It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.

GCP has some regions where all of the AZs are located in the same physical building. AFAIK they’ve stopped doing this in newer regions and have been migrating existing regions to have physically separate sites for each AZ.

AWS has always guaranteed that different AZs are physically separate sites, but that hasn’t always been the case with GCP.

>It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events

im only familiar with microsoft in this sense, but they have "geo-zone-redundant storage" which copies data across 3+ availability zones in the primary region, and also a secondary geographic region (i think at least 100 miles away if i remember, but maybe further). any incident would have to be massive to affect multiple geo-zones

They had a comparable problem in europe-west9 in 2023. Apparently they actually run into trouble if they lose a whole zone through a physical event. It's either thundering herd or simply insufficient capacity in the other two zones to take over all the load from the third zone.

Thundering heards during zonal outage is fairly common across the providers. You are substantially better off being multi-regional than multi-zonal.

Reply on news.ycombinator.com