Hacker News Reader

Comment by bashinator | Hacker News Reader

Never run prod in us-east-1.

I would argue the opposite. Run in us-east-1 if you want to have truly reliable infrastructure.

AWS cost saving pro tip: Save that 10c per minute Fault Injection Simulator charge by running in us-east-1...

Agree. 10s of millions in infrastructure running in us-east-1 here. Honestly it seems to be the only reliable way to teach people about failure and the importance of reserving capacity.

It's the one with the chaos monkey built in.

More like Chaos Gorilla. https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Gorill...

Kind of like Donkey Kong, but he's rolling flaming barrels down the hot aisles.

Some of us have to. Weirdly, the bitcoin exchanges are all there.

I see this a lot. Is there any reason why this particular region will have so many issues?

A big reason is it’s the largest region. More servers = more likely to have problems

EC2 existed in us-east-1 only for 2 years before it was launched in a second region.

Those 2 years were already enough to pick up some legacy cruft and special cases. us-east-1 is also the first region for various other old services, like s3 and route53.

I think it's also the largest region across many axis, which naturally means it will hit various scaling issues before other regions.

Not sure if it is related but it is usually one of the first to get or be trialing new features. I believe a few global services are run out of us-east-1 as well.

This is my understanding as well. New features are rolled out in us-east-1 first, then progressively through the remainder of the regions.

I remember being bitten by an RDS update, running in ap-southeast-2.

Took out a few of our production databases, seems like Australia is the beta testing region for at least a few services.

At least in the US you have the choice of other regions.

That joke is getting pretty tired by now and restating it seems unlikely to be helpful for anyone who's affected by the rate-limited API calls in question.

It's not a joke. The us-east-1 region is one of if not the oldest. Some of the infrastructure is aging because it's hard to phase out. Amazon has learned a lot since then and the newer regions have reaped all the benefits, where Virginia hasn't. There are components that exist in that region and don't exist in others but as a rule, I use those as remote resources and keep all my critical infra in Ohio, if I need to be near east.

Have you actually measured it? I’ve had production workloads in us-east-1 for over a decade and haven’t seen it be notably different than the other regions. There’ve been a handful of times where there’s been a truly regional outage (17 minutes of network connectivity in 2011) and a couple of broader service outages (S3, IAM), but it’s not like the other regions never have problems.

In general most of the time when I’ve heard people say “us-east-1 is unreliable” it’s ended up that they’re really telling you that they were in just one AZ, thought the RDS multiple AZ mode was too expensive, etc. and standard architectural guidance would have taken care of it.

Reply on news.ycombinator.com