AWS West Outage (US-WEST-2 region) | Hacker News Reader

AWS West Outage (US-WEST-2 region)

status.aws.amazon.com

57 pointsrabidonrails23 comments

Open articleView on HN

Seems to have hit pagerduty pretty hard... your poor on-call guy may have no idea his app is busted.

Interesting. I got a page during the incident, so at least some of it was working.

Pager as in - the electronic device that existed before the cellphone?

Paged as in alerted or summoned - this service https://www.pagerduty.com/

Pager as in - the electronic device that existed before the cellphone?

Everything is closed.

Direct connect: October 18, 2017 at 4:37:09 PM UTC-5

Internet connectivity: October 18, 2017 at 4:33:29 PM UTC-5

EC2 VPC network health: October 18, 2017 at 4:13:00 PM UTC-5

---

Personal Health Dashboard seems like its being udpated faster.

https://phd.aws.amazon.com/

---

Opened at: October 18, 2017 at 4:10:19 PM UTC-5

DirectConnect operational issue

02:10 PM PDT We are investigating network connectivity issues affecting Direct Connect customers using the US-WEST-2 Region.

---

Opened at: October 18, 2017 at 3:59:42 PM UTC-5

Network Connectivity

01:59 PM PDT We are investigating Network Connectivity issues in the US-WEST-2 Region.

---

Opened at: October 18, 2017 at 3:59:00 PM UTC-5

EC2 VPC network health internet issue

Beginning at Wed, 18 Oct 2017 20:59:00 GMT, some instances are experiencing elevated packet loss between the us-west-2a Availability Zone and the Internet. We are now investigating this issue.

Would they be publishing a post-mortem? If yes, where do I get to read it? Someone be kind enough to enlighten me. Sincerely,

Yes,if you can clear their hiring bar, post mortems for all such incidents are shared among AWS employees... Some of them are shared company wide

I am going to clear their hiring bar.

I just noticed Stripe's API returning intermittent 500 errors for the first time in forever. Now I know why! =)

yeah so did we.. and then also from auth0. aws status page was next thing to check.

These things happen to every datacentre. That’s why AWS has multiple, separately managed regions that you can use to keep your app up with.

and pay double? no way

Enjoy your downtime then.

Hardware will always fail at some point. You can either add extra resources elsewhere, or deal with that inevitable downtime. Both are reasonable, depending on the impact of downtime.

It's not paying double. Take the total number of instances serving the app and divide them across data centers and availability zones within datacenters. This might require adding a few of servers if the app is replicated across a very small number of instances, but for apps that already have a significant number of instances to handle the capacity, it's just a matter of rearranging things.

Depending on how quickly new instances of the app can be brought up to, it may be worthwhile to run a bit more capacity so that the remaining instances don't melt down should all the instances in one zone become unavailable.

That's why my aws console wasn't working properly this afternoon. It seems that Amazon is not using it's services correctly

Nonsense. The console is hosted out of us-east-1. Also, please learn to spell before posting. Thanks.

But AWS has the concept of availability zones that they equate to a data center. AWS refers to an AZ as an "isolated location". It's becoming clear with the S3 outages and this outage that isolation is not as robust as expected so now we see more reference to cross-region replication. But unlike cross-AZ services like S3 or DynamoDB, you need to build your own cross-region replication and failover.

Yep. My take: humans fail at more than hardware does, and AWS takes a lot of action per region.

Title says AWS West, but it's specifically us-west-2 region (Oregon).

We noticed less network connections around 1:30ish...

Yeah noticed this in our production app today. Thanks AWS. Thank goodness we had multi AZ DB running and other failovers.

Reply on news.ycombinator.com