For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...
Disclaimer: I work for Google Cloud.
If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1."
Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).
Also keep in mind GCP has a global network so it's possible some other regions are seeing disconnects/packet drops, though the source of the issue is contained to a single datacenter.
When did that change? I'm glad it did, but it seems like that was long in coming.
Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.
For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
That said, that's how Microsoft handles it, I'm not sure how GCP compares.
It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.
im only familiar with microsoft in this sense, but they have "geo-zone-redundant storage" which copies data across 3+ availability zones in the primary region, and also a secondary geographic region (i think at least 100 miles away if i remember, but maybe further). any incident would have to be massive to affect multiple geo-zones
AWS has always guaranteed that different AZs are physically separate sites, but that hasn’t always been the case with GCP.
If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.
(who would be motivated to check if anything is restricted to only us-east-1)
IAM is usually the culprit when us-east-1 issues affect other regions, but Lambda runs independently in every region.
here’s one third party, it was pretty well documented as a control plane failure in us-east-1
https://aws.amazon.com/message/101925/
This, too, was contained to us-east-1:
"While the event started at 11:48 PM PDT on October 19 and ended at 2:20 PM PDT on October 20, there were three distinct periods of impact to customer applications. First, between 11:48 PM on October 19 and 2:40 AM on October 20, Amazon DynamoDB experienced increased API error rates in the N. Virginia (us-east-1) Region. Second, between 5:30 AM and 2:09 PM on October 20, Network Load Balancer (NLB) experienced increased connection errors for some load balancers in the N. Virginia (us-east-1) Region. This was caused by health check failures in the NLB fleet, which resulted in increased connection errors on some NLBs. Third, between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed and, while instance launches began to succeed from 10:37 AM, some newly launched instances experienced connectivity issues which were resolved by 1:50 PM."
my original claim was that their control plane runs in us-east-1. this is a fact.
The claim I'm testing is whether the impact spread to AWS services in other regions.
> At this point we’re playing semantic nittery.
It's not "semantic nittery." It's important as an engineer to be precise about the impact of incidents and how systems are architected, as opposed to making broadly sweeping (and false) claims that "[an entire] control plane runs in us-east-1" and "the whole fucking internet went down that day."
As someone who worked for AWS, I have insider knowledge that what you said is not true as a blanket statement. They generally take great care to isolate both control and data planes regionally, with rare exceptions where service data has to be globally consistent. Moreover, they have always recommended not to rely on control planes to respond to incidents: https://docs.aws.amazon.com/wellarchitected/latest/framework...
Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.
As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
Do you blame them? You not only have to pay double for the infra but also be charged for inter-region data transfers. And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
Well, yeah. Redundancy isn't free. I don't see how this is controversial.
> And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
What do you mean, "IAM doesn't work"? IAM has always worked, even during control plane impairments. You cannot depend on control plane operations to work during incidents; you should plan in advance for failure and have your orchestration already set up so that you don't have to touch a thing when you have to weigh away from an AZ or region.
https://docs.aws.amazon.com/wellarchitected/latest/framework...
What do you mean, "IAM always worked"? The last failure was very recent, only a year ago[0]
[0] https://www.dataminded.com/resources/how-to-protect-against-...
As the page you linked to stated:
> IAM (Identity and Access Management) configuration was unavailable, preventing updates to roles and policies, but cached credentials allowed most running workloads to continue.
Perhaps I should have been clearer about what I meant about IAM still working. That's on me.
2. That article you linked to does not support your claim. That is a report of a single-zone incident impacting us-east-1 only. Five other AZs in the same region were still fully operational.
[1] These are not the same as control plane dependencies: you might not be able to make certain mutations, but everything should still work in the steady state.
One more than one occasion there are AWS URLs, including redirects inside the console, that include US East 1 in the URL that were only spotted during an outage. Manually changing it to an different region allowed us to get around it but to wholely blame customers when you don't know the context is bad form.
But there are also always exceptions. I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s. A real problem if things have scaled down overnight.
Yeah, that tracks. The Console has often been wonky like that.
> I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s
Do you mean scaling failed due to insufficient capacity, or something else? ICE errors I get (thundering herd of attempted capacity shifting); but otherwise, that sounds implausible, and I respectfully believe you are mistaken about cause and effect. Happy to evaluate a causal chain of evidence that proves you correct, though.
You are allowed to be wrong.
How do you know for certain that they are causally connected? Take me through it.
I will do better at saving exact details for internet strangers in future.