Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
aws.amazon.com
aws.amazon.com
We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and completely took down our data tier and caused a multi-hour outage, compounded by retries and thundering herds. Understanding this problem under the stressful scenario is extremely difficult and a harrowing experience. Anticipating this kind of issue is very very tricky.
Naive responses to this include "better testing", "we should be able to do this", "why is there no observability" etc. The problem isn't testing. Complex systems behave in complex ways, and its difficult to model and predict, especially when the inputs to the system aren't entirely under your control. Individual components are easy to understand, but when integrating, things get out of whack. I can't stress how difficult it is to model or even think about these systems, they're very very hard. Combined with this knowledge being distributed among many people, you're dealing with not only distributed systems, but also distributed people, which adds more difficulty in wrapping this around your head.
Outrage is the easy response. Empathy and learning is the valuable one. Hugs to the AWS team, and good learnings for everyone.
I'm outraged that AWS, as a company policy, continues to lie about the status of their systems during outages, making it hard for me to communicate to my stakeholders.
Empathy? For AWS? AWS is part a mega corporation that is closing in on 2 TRILLION dollars in market cap. It's not a person. I can empathize with individuals who work for AWS but it's weird to ask us to have empathy for a massive faceless, ruthless, relentless, multinational juggernaut.
I may be wrong, but I try to apply the https://en.m.wikipedia.org/wiki/Principle_of_charity
That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.
I think most people's experience with "VP's" makes them not realize what AWS VP's do.
VP's here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to "Call me when the data center is back up and running again!"
They are on the tech call, working with the engineers, evaluating the problem, gathering the customer impact, and attempting to balance communicating too early with being precise.
Is there room for improvement? Yes. I wish we would just throw up a generic "Shit's Fucked Up. We Don't Know Why Yet, But We're Working On It" message.
But the reason why we don't, doesn't have anything to do with having to get VP approval to put that message up. The VP's are there in the trenches most of the time.
For example, The meaning of "S3 was affected" is subject to a lot of interpretation. STS was down, which is a blocker for accessing S3. So, the end result is S3 is effectively down, but technically it is not. How does one convey this in a large org? You run S3, but not STS, it's not technically an S3 fault, but an integration fault across multiple services. If you say S3 is down, you're implying that the storage layer is down. But it's actually not. What's the best answer to make everyone happy here? I cant think of one.
A couple years ago all our services at our data center just vanished. I call the data center and they start creating a ticket. "Can you tell me if there is a data center outage?" "We are currently investigating and I don't have any information I can give you." "Listen, if this is a problem isolated to our cabinet, I need to get in the car. I'm trying to decide if I need to drive 60 miles in a blizzard."
That facility has been pretty good to us over a decade, but they were frustratingly tight-lipped about an entire room of the facility losing power because one of their power feeder lines was down.
Could AWS improve? Yes. Does avoiding AWS solve these sorts of problems? No.
Personally i am a believer in mixed environments, public webservers etc in the "cloud", locally used systems and backup "in house" with a second location (both in Data-centers or at least one), and no, i don't talk about the next google but the 99% of businesses.
My company was not affected by this outage because we are multi region. Cheapest and quickest option if you want to have at least some fault tolerance.
It is naive to assume people bashing AWS are uncapable to running things better, cheaper, faster, across many other vendors, on-prem, colocation or what not.
> Outrage is the easy response.
That is what made AWS get the marketshare it has now in the first place, the easy responses.
The main selling point of AWS in the beginning was "how easy is to sping a virtual machine". After basically every layman started recommending AWS and we flocked there, AWS started making things more complex than it should. Was that to make harder to get out of it? IDK.
> Empathy and learning is the valuable one.
When you run your infrastructure and something fails and you are not transparent, your users will bash you, independently who you are.
And that was another "easy response" used to drive companies towards AWS. We developers were echoing that "having a infrastructure team or person is not necessary", etc.
Now we are stuck in this learned helplessness where every outage is a complete disaster in terms of transparency, multiple services failing, even for multi-region and multi-az customers, we saying "this service here is also not working" and AWS simple states that service was fine, not affected, up and running.
If it was a sysadmin doing that, people will be asking for his/her neck with pitchforks.
I don’t think this is fair for a couple reasons:
1. AWS would have had to scale regardless just because of the number of customers. Even without adding features. This means many data centers, complex virtual networking, internal networks, etc. These are solving very real problems that happen when you have millions of virtual servers.
2. AWS hosts many large, complex systems like Netflix. Companies like Netflix are going to require more advanced features out of AWS, and this will result in more features being added. While this is added complexity, it’s also solving a customer problem.
My point is that complexity is inherent to the benefits of the platform.
The disdain I saw was towards those claiming that all you need is AWS, that AWS never goes down, and don't bother planning for what happens when AWS goes down.
AWS is an amazing accomplishment, but it's still a single point of failure. If you are a company relying on a single supplier and you don't have any backup plans for that supplier being unavailable, that is ridiculous and worthy of laughter.
Totally understand that complex systems behave in incomprehensible ways (hopefully only temporarily incomprehensible). But they're selling people on the idea of trading your complex system, for their far more complex system that they manage with such great expertise that it is more reliable.
They have SLAs. And there are clauses that cover the weird edge cases for when the SLAs are not met.
I remember my first experience realizing the client retry logic we had implemented was making our lives way worse. Not sure if it's heartening or disheartening that this was part of the issue here.
Our mistake was resetting the exponential backoff delay whenever a client successfully connected and received a response. At the time a percentage but not all responses were degraded and extremely slow, and the request that checked the connection was not. So a client would time out, retry for a while, backing off exponentially, eventually successfully reconnect and then after a subsequent failure start aggressively trying again. System dynamics are hard.
And have to be actually tested. Most of them are designs based on nothing but uninformed intuition. There is an art to back pressure and keeping pipelines optimally utilized. Queueing doesn’t work like you think until you really know.
It is something which needs to be designed and tuned in place and evades design "getting it right" without real world feedback.
And you also simply have to reach a certain somewhat large scale for it to matter at all, the amount of excess capacity you have because of the available granularity of capacity at smaller scales eats up most of the need for it and you can get away with wasting a bit of money on extra scale to get rid of it.
It is also sensitive to small changes so textbook examples might be implemented wrong with one small detail that won't show itself until a critical failure is happening.
It is usually the location of the highest complexity interaction in a business infrastructure which is not easily distilled to a formula. (and most people just aren't educationally prepared for nonlinear dynamics)
During my studies we had a course where we built a distributed system and had to model it's performance mathematically. It was really hard to get the model to match the reality and vice-versa. So many details are hidden in a library, framework or network adapter somewhere (e.g buffers or things like packet fragmentation).
We used the book "The Art of Computer Systems Performance Analysis" (R. Jain), but I don't recommend it. At least not the 1st edition which had a frustrating amount of serious, experiment-ruining errata.
1) airplanes: they crashed, _a lot_. We used data recorders and stringent process to make air travel safety commonplace.
2) cars: so many accidents accident research. The solution comes after the disaster.
3) large buildings and structures: again, the master work of time, attempts, failures, research and solutions.
If we really want to get serious about this (and I think we do) we need to stop reinventing infrastructure every 10 years and start doubling down on stability. Cloud computing, in earnest, has only been around a short while. I’m not even convinced it’s the right path forward, just happens to align best with business interests, but it seems to be the devil we’re stuck with so now we need to really dig in and make it solid. I think we’re actually in that process right now.
Also, experiments at this size and speed have never been carried out before.
And statistical behaviours are very difficult to understand. First thing: 99.9999% uptime for ALL users is HUGELY different from 99.999%.
As a matter of fact, this was just one of amazon’s zones, rememeber.
Edit: finally, the right model for these systems might well have no mean (fat tails…) and then where do the statistics go from there?
Or because they read it on a Google|AWS Engineering blog
Or tell them directly that "We have screwed up. The service is currently overloaded. Thank you for your patience. If you still haven't given up on us, try again a less busy time of day. We are very sorry."
There are several options, and finding the best one depends a bit on estimating the behaviour of your specific target audience.
We've seen plenty of S3 errors during that period. Kind of undermines credibility of this report.
> For example, while running EC2 instances were unaffected by this event
That ignores the 3-5% drop in traffic I saw in us-east-1 on EC2 instances that only talk to peers on the Internet with TCP/IP during this event.
Perhaps that distinction has value if your workloads did not depend on network connectivity externally for example say S3 access without vpc and only compute some DS/ ML jobs perhaps.
I was getting 503 "service unavailable" from STS during the outage most of the time I tried calling it.
I guess by "elevated latency", they mean from anyone with retry logic that would keep trying after many consecutive attempts?
AWS in general is trying hard to do the right thing for customers, and obviously has a long ways to go. But man, a few specific orgs have some frustrating holdover policies.
You are responding to a comment that suggests they're misrepresenting the truth (which wouldn't be the first time even in last few days) in communication to their customers.
As always, they are doing the right thing for themselves only.
EDIT: I think that you should mention being an Engineer at Amazon AWS in your comment.
They treat 5xx errors as non-errors but this is not the case with rest of the world. "Increased timeouts" is Amazon's untruthful term for "not working at all".
Disruption of the standard incident response mechanism seems to be a common element of longer lasting incidents.
I don’t remember the exact details, but it was something along the lines of a config change went out that caused systems to incorrectly assume there were huge bandwidth constraints. Load shedding kicked in to drop lower priority traffic which ironically included monitoring data rendering GCP responders blind and causing StackDriver to go blank for customers.
Doesn't the region going down mean that _all_ its AZs have gone down? Or is my mental model of this incorrect?
A region is a networking paradigm. An AZ is a group of 2-6 data centers in the same city more or less.
If a region goes down or is otherwise impacted, its AZs are unavailable or similar.
If an AZ goes down, your VMs in said centers are disrupted in the most direct sense.
It's the difference between loss of service and actual data loss.
So when failures happen in that region (and they happen more commonly than others due to age, scale, complexity) then they can be globally impacting.
Availability zones are supposed to be another fault boundary, and things are generally pretty solid, but every so often problems spill over when they shouldn't.
The general impression I get is that us-east-1's issues tend to stem from it being singularly huge.
(Source: Work at AWS.)
Literally all our AWS resources are in EU/UK regions - and they all continued functioning just fine - but we couldn't sign in to our AWS console to manage said resources.
Thankfully the outage didn't impact our production systems at all, but our inability to access said console was quite alarming to say the least.
These big outages are noteworthy because they _do_ affect people who correctly architected for reliability — and they're pretty rare. This one didn't affect one of my big sites at all; the other was affected by the S3 / Fargate issues but the last time that happened was 2017.
That certainly could be better but so far it hasn't been enough to be worth the massive cost increase of using multiple providers, especially if you can have some basic functionality provided by a CDN when the origin is down (true for the kinds of projects I work on). GCP and Azure have had their share of extended outages, too, so most of the major providers tend to be careful to cast stones about reliability, and it's _much_ better than the median IT department can offer.
“If you're having SLA problems I feel bad for you son I got two 9 problems cuz of us-east-1”
If you wanted to deploy something similar, like Cassandra across AZs, or even regions you're welcome to do that. But now you're on the hook for the availability of the system. Are you going to get higher availability running your own Cassandra implementation than the DynamoDB team? Maybe. DynamoDB had a pretty big outage in 2015 I think. But that's a lot more work than just using DynamoDB IMO.
this is both more and less true than you might think. for most regional endpoints teams leverage load balancers that are scoped zonally, such that ip0 will point at instances in zone a, ip1 will point at instances in zone b, and so on. Similarly, teams who operate "regional" endpoints will generally deploy "zonal" environments, such that in the event of a bad code deploy they can fail away that zone for customers.
that being said, these mitigations still don't stop regional poison pills or otherwise from infecting other AZs unless the service is architected to zonally internally.
[1] https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B...
[0]: https://aws.amazon.com/blogs/aws/in-the-works-aws-canada-wes...
Our SOP is to cut over to a second region the moment we see any AZ-level shenanigans. We've been burned too often.
Usually there would be high network error rates which were usually enough to make RDS Postgres fail over if it was in the impacted AZ
The only real "outage" was DNS having extremely high error rates in a single us-east-1 AZ to the point most things there were barely working
Lack of instance capacity, especially spot, especially for the NVMe types was common of CI (it used ASGs for builder nodes). It'd be pretty common for a single AZ to run out of spot instance types--especially the NVMe ([a-z]#d types)
My system is latency and downtime tolerant, but I’m thinking I should move all my Kafka processing over to us-west-2
First, if you can, avoid us-east-1. Yes, you’ll miss new features, but it’s also the least stable region.
Second, go multi AZ for production workloads. Safety of your customer’s data is your ethical responsibility. Protect it, back it up, keep it as generally available as is reasonable.
Third, you’re gonna go down when the cloud goes down. Not much use getting overly bent out of shape. You can reduce your exposure by just using their core systems (EC2, S3, SQS, LBs, Cloudfrount, RDS, Elasticache). The more systems you use, the less reliable things will be. However, running your own key value store, api gateway, event bud, etc., can also be way less reliable than using their’s. So, realize it’s an operational trade off.
Degradation of your app / platform is more likely to come from you than AWS. You’re gonna roll out bad code, break your own infra, overload your own system, way more often than Amazon is gonna go down. If reliability matters to you, start by examining your own practices first before thinking things like multi region or super durable highly replicated systems.
This stuff is hard. It’s hard for Amazon engineers. Hard for platform folks at small and mega companies. It’s just, hard. When your app goes down, and so does Disney plus, take some solace that Disney in all their buckets of cash also couldn’t avoid the issue.
And, finally, hold cloud providers accountable. If they’re unstable and not providing service you expect, leave. We’ve got tons of great options these days, especially if you don’t care about proprietary solutions.
Good luck y’all!
AWS (and others) make egress costs insanely expensive for any startup to consider leaving with their data, also there is constant push to either not support open protocols or extend /expand them in ways making it hard to migrate a code base easily.
If the advise is to use only effectively use managed open source components then why AWS at all ? most competent mid sized teams can do that much cheaper with a colo providers like OVH/hetzner.
The point of investing in AWS is not outsource running base infra, if we should stay away from leveraging the kind of cloud native services us mere mortals cannot hope to build or maintain.
Also this avoid us-east-1 advice is bit frustrating, AWS does not have to experiment with new services always in the same region,it is not marked as experimental region or has reduced SLAs , if it is inferior/preview/beta than call it out in the UI and contract, what about when there is no choice? If cloudfront is managed in us-east-1 and we shouldnt now use it ? Why use the cloud then ?
if your engineering only discovers scale problems at us-east-1 along with customers perhaps something is wrong ? aws could limit new instances in that region and spread the load, playing with customers like this who are at your mercy just because you can is not nice.
Disney can afford to go down, or build their cloud, small companies don't have deep pockets to do either
I have seen this repeated many times, but don't understand it. Yes egress is expensive, but they are not THAT expensive compared to storage. S3 egress per GB is no more than 3x the price of storage, i.e. moving out just cost 3 month of storage cost (there's also API cost but that's not the one often mentioned).
Is egress pricing being a lock-in factor just a myth? Is there some other AWS cost I'm missing? Obviously there will be big architectural and engineering cost to move, but that's just part of life.
But really, egress costs aren’t locking you in. It’s the hard coded AWS apis, terraform scripts and technical debt. Having to change all of that and refactor and reoptimize to a different providers infrastructure is a huge endeavor. That time spent night have a higher ROI being put elsewhere
At scale when costs matter, you would have lifecycle policy tuned to your needs taking advantage of these classes. Any typical production workload is hardly paying only S3 base price for all/most of its storage needs, they will have mix of all these too.
[1] if there is substantial data in glacier regular, the costing completely blows through the roof, retrieval +egress makes it infeasible unless you activily hate AWS enough to spend that kind of money
Don't lock yourself in.
If your use only services built and managed by your docker images why use the cloud in the first place ? It would be cheaper to host on a smaller vendor , the reliability is not substantially better with big cloud than tier two vendors, that difference between say OVH and AWS is not that valuable to most applications to be worth the premium.
In IMO, if you don't leverage cloud native services offered by GCP or AWS then cloud is not adding much value to your stack.
A wait for X provider to fix it for you situation is infinitely more stressful than an 'I have played myself, I will now take action' situation.
Situations out of your (immediate) resolution control feel infinitely worse, even if the customer impact in practice of your fault vs cloud fault is the same.
Not necessarily. You just need to not be stuck with a single cloud provider. The likelihood of more than one availability zone going down on a single cloud provider is not that low in practice. Especially when the problem is a software bug.
The likelihood of AWS, Azure, and OVH going down at the same time is low. So if you need to stay online if AWS fail, don't put all your eggs in the AWS basket.
That means not using proprietary cloud solutions from a single cloud provider, it has a cost so it's not always worth it.
but perhaps some software design choices could be made to alleviate these costs. For example, you could have a read-only replica on azure or whatever backup cloud provider, and design your software interfaces to allow the use of such read only replicas - at least you'd be degraded rather than unavailable. Ditto with web servers etc.
This has a cost, but it's lower than entirely replicating all of the proprietary features in a different cloud.
The more complexity you build into your own systems on top of the providers you depend on, the more likely you are to shoot yourself in the foot when you run into complexity issues that you’ve never seen before.
And the times that is most likely to happen is when one of your complex service providers goes down.
If the kind of thing you’re talking about could be feasibly done, then Netflix would have already done it. The fact that Netflix hasn’t solved this problem is a strong indicator that piling more proprietary complexity on top of all the vendor complexity you inherit from using a given service, well that’s a really hard problem in and of itself.
Don't miss the point of being able to do something about it instead of multi hours outage and being in the dark regarding what is going on.
Easy to say, but difficult to do in practice (leaving a cloud provider)
Just because it takes years doesn’t meant it can’t happen.
Ugh. I have a hard time with this one. Back in the day, EBS had some really awful failures and degradations. Building a greenfield stack that specifically avoided EBS and stayed up when everyone else was down during another mass EBS failure felt marvelous. It was an obvious avoidable hazard.
It doesn't mean "avoid EBS" is good advice for the decade to follow, but accepting failure fatalistically doesn't feel right either.
At this point, it’s reliable enough that even if it were to go down, it’s more safe than not using it. I’d put EBS in the pantheon of “core” services I never mind using these days.
Geez. We have decades-old cloud war stories now? I suddenly feel really old.
> Third, you’re gonna go down when the cloud goes down. Not much use getting overly bent out of shape.
“Whoops, our provider is down, sorry!” is not taking responsibility with customer data at all.
We’re not discussing data loss in this thread specifically. This is about a couple of hours of downtime per year.
This seems pretty obviously false to me. My company has several EKS clusters in us-east-1 with most of our workloads running on Fargate. All of our Fargate pods were killed and were unable to be restarted during this event.
Either way, was surprising to us that already provisioned compute was impacted.
Couldn't just delete the Fargate profile without a working EKS control plane. I lucked out in that the label selector the kube-dns Service used was disjoint from the one I'd set in the Fargate profile, so I just made a new "coredns-emergency" deployment and cluster networking came back. (cluster-autoscaler was moot since we couldn't launch instances anyway.)
I was hoping to see something about that in this announcement, since the loss of live pods is nasty. Not inclined to rely on Fargate going forward. It is curious that you saw those pod ages; maybe Fargate kubelets communicate with EKS over the AWS internal network?
Obviously networking issues can cause any number of symptoms but it seems like an unusual detail to leave out to me. Unless it was another ongoing outage happening at the same time.
Permission denied is something altogether because it implies the request reached an authorisation system, was evaluated and denied.
No, absolutely not. That's why I'm on AWS.
But what we are all ACTUALLY complaining about is ongoing lack of transparent and honest communications during outages and, clearly, in their postmortems.
Honest communications? Yeah, I'm pretty sure I could do that much better than AWS.
What does this even mean ? I bet most people use DynamoDB via a VPC, in a Lambda or in EC2
VPC Endpoints create a DynamoDB endpoint in your VPC, from the documentation:
"When you create a VPC endpoint for DynamoDB, any requests to a DynamoDB endpoint within the Region (for example, dynamodb.us-west-2.amazonaws.com) are routed to a private DynamoDB endpoint within the Amazon network. You don't need to modify your applications running on EC2 instances in your VPC. The endpoint name remains the same, but the route to DynamoDB stays entirely within the Amazon network, and does not access the public internet."
If you want to dive into postmortems, there are some repos linking other examples
Not sure if any AWS support staff are monitoring this thread, but the article said:
> Customers also experienced login failures to the AWS Console in the impacted region during the event.
All our AWS instances / resources are in EU/UK availability zones, and yet we couldn't access our console either.
Thankfully none of our instances were affected by the outage, but our inability to access the console was quite worrying.
Any idea why this was this case?
Any suggestions to mitigate this risk in the event of a future outage would be appreciated.
- https://github.com/Netflix/Hystrix - https://www.youtube.com/watch?v=CZ3wIuvmHeM I think talks about Hystrix a bit, but I'm not sure if it's the presentation I'm thinking of from years ago or not.
Just curious, is this scaling an AWS job or a client job? Looks like an AWS one from the context. I'm wondering if they are deploying additional data centers or something else?
The reality is that there’s a handful of people in the world that can operate systems at this sheer scale and complexity and I have mad respect for those in that camp.
Say what you will, but I can automate a status dashboard in a couple days--yes, even at AWS scale.
No reason the dashboard should be green for hours while their engineers and support are aware things aren't working.
What you can do at what you think is AWS scale has no bearing on what you could actually do at real AWS scale.
The point of eating at a restaurant is that I can't/don't want to cook. Likewise, I use AWS because I want them to do the hard work and I'm willing to pay for it.
How does that abrogate my right to complain if it goes badly (regardless of whether I could/couldn't do it myself)?
Apache Solr
Apache Druid
Apache Flink
ElasticSearch
Flume
Apache Dubbo
Logstash
Kafka
That list comes from: https://unit42.paloaltonetworks.com/apache-log4j-vulnerabili...
The attack surface is quite a bit larger than many realize. I recently had a conversation with a person who wasn't at a Java shop so wasn't worried... until he said "oh, wait, ElasticSearch is vulnerable too?"
You'll even see it in things like the connector between CouchBase and ElasticSearch ( https://forums.couchbase.com/t/ann-elasticsearch-connector-4... ).
Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by their "green board" is meaningless. During the event they were late and reluctant to report it and its significance. My point is that they are wrongly incentivized to keep the board green at all costs.
They "own up to it" in their postmortems, but after multiple failures they're still unwilling to implement the obvious solution and what is widely regarded as best practice: host the status page on a different platform.
This seems like bad faith to me based on my experience when I worked for AWS. As they repeated many times at Re:Invent last week, they've been doing this for 15+ years. I distinctly remember seeing banners like "Don't update the dashboard without approval from <importantSVP>" on various service team runbooks. They tried not to say it out loud, but there was very much a top-down mandate for service teams to make the dashboard "look green" by:
1. Actually improving availability (this one is fair).
2. Using the "Green-I" icon rather than the blue, orange, or red icons whenever possible.
3. They built out the "Personal Health Dashboard" so they can post about many issues in there, without having to acknowledge it publicly.
It's smart politics -- I don't blame them but I don't trust the dashboard either. There's established patterns now of the AWS dashboard being useless.
If I want to check if Amazon is down I'm checking Twitter and HN. Not bad faith -- no faith.
Twice is a coincidence.
Three times is a pattern.
But this… This is every time.
The problem is that the dashboard requires VP approval to be updated. Which is broken. The dashboard should be automatic. The dashboard should update before even a single member of the AWS team knows there's something wrong.
Let's not act like this is the first time this has happened. It's bad faith that they do not change when their promise is they hire the best to handle infrastructure so you don't have to. It's clearly not the case. Between this and billing I we can easily lay blame and acknowledge lies.
This to me is really bad. Even as a small company, we keep our support infrastructure separate. For a company of Amazon's size, this is a shitty excuse. If I cannot even reach you as a customer for almost 7 hours, that is just nuts. AWS must do better here.
Also, is it true that the outage/status pages are manually updated ? If yes, there is no excuse why it was green for that long. If you are manually updating it, please update asap.
there's such as thing as too much dogfooding.
It turned out our call center supplier had something running on AWS, and it took out our entire phone system. After this situation settles, I'm tempted to ask my supplier to see what they're doing to get around this in the future, but I doubt even they knew that AWS was used further downstream.
AWS operates a lot like Amazon.com, the marketplace now--you can try to escape it, but it's near impossible. If you want to ban usage of Amazon's services, you're going to find some service (AWS) or even a Shopify site (FBA warehouse) who uses it.
One of our auxiliary services that's basically a pass through to AWS was offline nearly the entire day. Yet, this communication doesn't even mention that fact. In fact, it almost tries to suggest the opposite.
Likewise, AWS is reporting S3 didn't have issues. Yet, for a period of time, S3 was erroring out frequently because it was responding so slowly.
SLAs that refund only the cost of the individual service that was down is worthless.
SLAs that require separate proof and refund requests for each and every service that was affected are nearly worthless.
There needs to be an independent body set up by a large cloud customers to monitor availability and enforce refunds.
Unless you're talking about some board other than the Service Health Dashboard, this isn't true. They dropped EC2 down to degraded pretty early on. I bemusedly noted in our corporate Slack that every time I refreshed the SHD, another service was listed as degraded. Then they added the giant banner at the top. Their slight delay in updating the SHD at the beginning of the outage is mentioned in the article. It was absolutely not all green for the duration of the outage.
Sagemaker, for example, was down all day. I was dead in the water on a modeling project that required GPUs. It relied on EC2, but nobody there even thought to update the status? WTF. This is clearly executives incentivized to let a bug persist. This is because the bug is actually a feature for misleading customers and maximizing profits.
They are barely doing better than 2 9s.
My boss had not given me any special directions here so I assumed I was supposed to do this honestly. I set our status as "Red" and then listed out what were, I felt, quite compelling reasons to think we were Red. The gist of it was that our velocity was negative. More work items were getting created and assigned to us than we closed, and we still had high priority items open from previous dates. There was zero chance, in my estimation, that we would meet our deadlines, so I called us Red.
This did not go over well. Everyone at the Launch Readiness meeting got mad at me for declaring Red. Our VP scolded me in front of the entire meeting and lectured me about how I could not unilaterally declare our team red. Her logic was, if our team was Red, that meant the entire project was Red, and I was in no position to make that call. Other managers at the meeting got mad at me too because they felt my call made them look bad. For the rest of my manager's absence I had to first check in with a different manager and show him my Launch Readiness status and get him to approve my update before I was allowed to show it to the rest of the group.
For the rest of the time that I went to Launch Readiness I was forbidden from declaring Red regardless of what our metrics said. Our team was Yellow or Green, period.
Naturally, we wound up being over a year late on the deadlines, because, despite what they compelled us to say in those meetings, we weren't actually getting the needed work done. Constant "schedule slips" and adjustments. Endless wasted time in meetings trying to rework schedules that would instantly get blown up again. Hugely frustrating. Still slightly bitter about it.
Anyway, I guess all this is to say that it doesn't surprise me that Amazon is bad about declaring Red, Yellow, or Green in other places too. Probably there is a guy in charge of updating those dashboards who is forbidden from changing them unless they get approval from some high level person and that person will categorically refuse regardless of the evidence because they want the indicators to be Green.
In essence - what you were saying is your boss lied the whole time, because how does one go from a presumed positive velocity to negative velocity in a week?
Additionally assuming you're a dev lead, it's a little surprising that this is your first meeting of this sorts. As dev lead, I didn't always attend them but my input is always sought on the status.
Sounds like you had a bad manager, and Amazon is filled with them.
1) If you keep status green for 5 years, while not delivering anything, the reality is the folks at the very top (who can come and go) just look at these colors and don't really get into the project UNLESS you say you are red :)
2) Within 1-2 years there is always going to be some excuse for WHY you are late (people changes, scope tweaks, new things to worry about, covid etc)
3) Finally you are 3 years late, but you are launching. Well, the launch overshadows the lateness. Ie, you were green, then you launched, that's all the VP really sees sometime.
Equipment was constantly breaking down, in ways that ranged from inconvenient to potentially dangerous. Seemingly basic design decisions, like the shape of chutes, were screwed up in mind-boggling ways (they put a right-angle corner partway down each chute, which caused packages to get stuck in the chutes constantly). We were short on equipment almost every day; things like poles to help us un-jam packages were in short supply, even though we could move hundreds of thousands of packages a day. On top of all this, the facility opened with half its sorting equipment, and despite promises that we'd be able to add the rest of the equipment in the summer, during Amazon's slow season...it took them two years to even get started.
And all the while, they demanded ever-increasing package quotas. At first, 120,000 packages/day was enough to raise eyebrows--we broke records on a daily basis in our first holiday rush--but then, they started wanting 200,000, then 400,000. Eventually it came out that the building wouldn't even be breaking even until it hit something like 500,000.
As we scaled up, things got even worse. None of the improvements that workers suggested to management were used, to my knowledge, even simple things like adding an indicator light to freight elevators.
Meanwhile, it eventually became clear that there wasn't enough space to store cargo containers in the building. 737s and the like store packages mostly in these giant curved cargo containers, and we needed them to be locked in place while working around/in them...except that, surprise, the people planning the building hadn't planned any holding areas for containers that weren't in use! We ended up sticking them in the middle of the work area.
Which pissed off the upper management when they visited. Their decision? Stop doing it. Are we getting more storage space for the cans? No. Are we getting more workers on the airplane ramp so we can put these cans outside faster? No. But we're not allowed to store those cans in the middle of the work area anymore, even if there aren't any open stations with working locks. Oh, by the way, the locking mechanisms that hold the cans in place started to break down, and to my knowledge they never actually fixed any of the locks. (A guy from their safety team claims they've fixed like 80 or 90 of the stations since the building opened, but none of the broken locks I've seen were fixed in the 2 years I worked there.)
In organisations with 100s of in-flight projects, it’s understandable that red is reserved for projects that are causing extremely serious issues right now. Otherwise, so many projects would be red, that you’d need a new colour.
Jeez, how many meetings did you go to, and how long was this person's vacation? I'm jelly of being allowed to take that much time off continuously.
"How dare you threaten our launch readiness go/no-go?!"
> While AWS customer workloads were not directly impacted from the internal networking issues described above, the networking issues caused impact to a number of AWS Services which in turn impacted customers using these service capabilities. Because the main AWS network was not affected, some customer applications which did not rely on these capabilities only experienced minimal impact from this event.
Yeah so vanilla LB and EC2 with no autoscaling were fine. Anyone using "serverless" or managed services had a real bad day
Bullshit someone at Amazon doesn’t have time to update the status.
Edit: not to mention APIGW and Cloudwatch APIs were down too.
if ($pain > $gain) {
move_your_shit_and_exit_aws();
}
sub move_your_shit_and_exit_aws
{
printf("Dude. We have too much pain. Start moving\n");
printf("Yeah. That won't happen, so who cares\n");
exit(1);
}We have an environment we have access to for hosting webpages for one of the highest leaders in the whole Dept of Navy. This environment was DOWN (not "degrade availability" or "high latencies"), literally off of the Internet entirely, for CONSECUTIVE WEEKS earlier this year.
Completely incommunicado as well. It just happened to start working again one day. We collectively shrugged our shoulders and resumed updating our part of it.
This is an outlier example but even our normal sites I would classify as 1 "nine" of availability at best.
#gatekeep your real availability metrics
#gaslight your customers with increased error rates
#girlboss
So was this in service to something like DynamoDB or some other service?
As in, did some of those extra services that AWS offers for lockin (and that undermines open source projects with embrace and extend) bomb the mainline EC2 service?
Because this kind of smacks of "Microsoft Hidden APIs" that office got to use against other competitors. Does AWS use "special hardware capabilites" to compete against other companies offering roughtly the same service?
They also recently announced Graviton ARM CPUs
So many overload issues seem to be caused by a single client, in a case where the right prioritization or rate limit rule could have contained any outage, but such a rule either wasn't in place or wasn't the right one due to the difficulty of knowing how to prioritize hundreds of clients.
Using more bandwidth or requests than yesterday should then be handled as capacity allows, possibly with a manual configured priority list, cap, or ratio. But "what I used yesterday" should always be served first. That way, any outage is contained to clients acting differently to yesterday, even if the config isn't perfect.
Then again they get like $100B in revenue that should buy some decent unit tests.
The most important services get the most attention from leaders who apply the most pressure, especially in the first ~2y of a fast-growing or high-potential product. So people skip tests.
Load shedding never kicked in, so things had to be shutdown for a bit and then restarted.
Seems their normal operating state might be what is called "meta-stable" - dynamically stable at a high thru-put (edited) unless/until a brief glitch bumps the system into the low work being finished state which is also stable.
I am sad to say, I find issues like this any time I look at retry logic written by anyone I have not interacted with previously on the topic. It is shockingly common even in companies where networking is their bread and butter.
I just find it amusing that they describe their back-off behaviors as "well tested" and in the same sentence, say it didn't back off adequately.
It must have been crazy hard to troubleshoot when you are flying blind because all your monitoring is unresponsive. Clearly more isolation with clearly delineated information exchange points are needed.
I’m not saying I fully disagree. But consolidation of the worlds hosting necessitates a very complicated platform and these things will happen, either due to that complexity, failures that can’t be foreseen or good old fashioned Sod’s law.
I know AWS marketing wants you to believe it’s all magic and rainbows, but it’s still computers.
That is an interesting way to phrase that. A 'well-tested' method, but 'latent issues'. That would imply the 'well-tested' part was not as well-tested as it needed to be. I guess 'latent issue' is the new 'bug'.
I saw in the Reddit thread someone mentioned all services that auth to other services on the backend were effected (not sure how truthful it is but that certainly made sense)
I guess this is why it took ages for the status page to update. They didn't know which things to turn red.
Very detailed.
The better way is, we should have two queues, one for valid messages and one for invalid messages.
(Source: Work at AWS.)
What was wrong with "affect"?
Of course it was DNS.
It is always* DNS.
Sounds like they lost the ability to update the dashboard. HN comments at the time were theorizing it wasn't being updated due to bad policies (need CEO approval) etc. Didn't even occur to me that it might be stuck in green mode.
In any case, AWS not updating their dashboard is almost a meme by now. Even for global service outages the best you will get is a yellow.
Having DNS problems sounds a lot like the Facebook outage of 2021-10-04. https://en.wikipedia.org/wiki/2021_Facebook_outage