Hacker News Reader

Comment by Sebb767 | Hacker News Reader

This seems like a major global outage. Good luck to those poor sysadmins over at Amazon, this must easily be in the millions lost per minute.

Let's see whether this outage takes out HN with it once again ;-)

View on HN

Well they can't blame AWS. AWS status pages shows no issues. (Although it never does even when there are)

AWS and Amazon run on completely separate systems/hardware.

Amazon's internal, last i read, was years behind AWS's infrastructure and they're generally jealous of not having the same tools available.

I tried finding the article, but maybe someone else can help me remember?

There was apparently always some kind of mix (AWS did start from internal requirements, though it quickly diverged). The OG services like S3 and DynamoDB power a lot of Amazon for sure. There’s been more bed of Amazon moving to AWS, most recently talk about shifting their main DBs.

That hasn't been 100% true for like four years. Most new things in Retail are built on AWS.

Are you certain? As of last year, Amazon was switching to Fastly for their retain images:

https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...

Some retail is definitely on AWS. Source: I work at AWS.

A few years back Amazon did a party for moving their warehouses off Oracle databases and into AWS Aurora, so that's not entirely true

This is incorrect.

https://aws.amazon.com/blogs/aws/prime-day-2021-two-chart-to...

I actually tried shopping on Amazon.com from AWS just to see if it was something local to me. Nope! I couldn't see any products or add them to my cart from AWS.

Can you normally do this?

Yes. I can ssh into an AWS instance and run Firefox with a remote X display. whatismyipaddress.com shows that I'm in the AWS datacenter.

I can also run a WireGuard VPN with AWS as the endpoint. The second way works better on slow connections.

Amazon uses Fastly instead of AWS at a lot of their product pages:

https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...

That’s just CDN, not the real service.

Well, AWS does have a strong CDN offering called Cloudfront so it is surprising.

My understanding is that Fastly has a lot more edge locations than AWS, so using their image CDN results in lower latencies for many users than they could achieve with Cloudfront alone.

It's not edge location count that matters, but Cloudfront doesn't use BGP Anycast but rather does a more traditional DNS-based routing and tries to spread the requests across multiple edge locations (even those farther away) for redundancy, intentionally.

When I asked for detail about why they don't use Anycast, the Cloudfront engineering team basically said their customers care more about uptime than latency and that full Anycast was too sketchy. Apparently amazon.com disagrees, at least. I'm also happy getting much lower first page view latency out of Cloudflare.

The one Amazon property that seems unaffected is their China site (https://amazon.cn). No doubt its running entirely separate infrastructure, unlike every other country's local Amazon site.

Example: https://www.amazon.cn/any-random-string-works-here/dp/032199...

Either everything is back, or amazon.com.au is also an exception.

Wouldn't be surprised if they serve .au from China

Why? They have a lot of infrastructure in Sydney, that would make no sense.

An interesting thing about Amazon outages like this is that most people wait and come back and make the purchase.

Everyone else will still make me wait several days or make me go to a store.

A lot of people who would have made impulse purchases don't though, once they have had time to reconsider.

Right but when they return how do you distinguish between a new impulse and a continuation of the old? This is one of those cases where marketers and businesses love to act like they know everything when in fact it's a huge info blind spot.

There's probably a lot less going to landfill today. That said, maybe more getting thrashed too:

https://www.itv.com/news/2021-06-21/amazon-destroying-millio...

Not too surprising I think... for myself and I suspect many/most, the calculus is something like:

A) check back in half an hour when Amazon will probably be working again

B) (1) find another online retailer which sells the kind of stuff I'm looking for (2) get the specific items I want in my cart (3) find my credit card (4) make an account (5) re-enter all my shipping information (6) shell out extra for shipping (7) repeat if I could not find a single retailer that sells all the items I wanted (8) curse myself the next day for committing to a retailer with slower shipping, unfamiliar returns process, etc., because I was too impatient to wait an hour for Amazon to be back up.

Not all of the steps in B apply to everyone of course. But it only takes a couple to make option A the winner, which I'd hazard is the case for many many Amazon shoppers.

There's an element to slow shipping that has less effect today, since we're starting the week, it doesn't matter if things turn up tomorrow or Friday if it is for weekend DIY projects.

I wouldn't mind using another retailer today and near enough forget about it until the weekend. If by some chance it doesn't arrive until the following Tuesday, it doesn't matter, I'll find something else to do at the weekend quickly enough!

However, if you need it pronto, I'd rather pay for a courier.

Many people have grown up with Amazon, and in the early day it looked like they were undercutting retail from top to bottom, making the highstreet look like the bad guys. As their warehouses get exposed, and the issues of no union appear, I'm less likely to buy from Amazon as I think there are ethics involved that need consideration.

Do I really want to buy something knowing that a poor soul was probably operating at 98%, just below maximum threshold before collapse to get this item boxed?

Ok so, option C: philosophize for a while, then follow option B. For the vast majority of customers, the calculus during an Amazon outage still favors option A.

More likely millions per hour. But even then it's not like all those sales will be lost. Most people will probably just come back later to make their purchase.

I d be interested to know the numbers - with the amount of impulse buys or marketing driven links, there s probably a lot of ads being paid for that redirect to nothing a user might just give up on

It's back

I think peak Christmas time shopping was a couple million per hour back in 2006.

I know we used to contrast that to million per minute (or second) kinds of outages that NYC financial firms could have (along with the SEC escorting you out in handcuffs if you really screwed up).

Yeah, but it's been 15 years. Amazon traffic has probably grown since then, most likely a lot.

Do you think that might have been why I specifically cited the year in my comment?

I don't think Amazon outages have ever taken HN down (HN doesn't run on AWS). We've done it to ourselves a few times though.

This was meant as a friendly nod to the fact that FAANG outages usually lead to a lot of people checking HN and generating insane amounts of load, like the time a few weeks ago when YouTube/Google Auth was down and took HN right with it :)

But overall this site is extremely well run, so please don't take this as more than a joke.

No worries! But I don't think this happened?

> YouTube/Google Auth was down and took HN right with it

Have I forgotten something obvious?

I think it was this [0] event. Some people commented on HN being overloaded [1], including me [2] :-) It was not completely down (iirc), but there were a lot of "Sorry, we cannot serve your request right now".

[0] https://news.ycombinator.com/item?id=25415989

[1] https://news.ycombinator.com/item?id=25416390

[2] https://news.ycombinator.com/item?id=25416144

yeah, Amazon is past the stage where they had sysadmins. They now have armies of Operations People with SDEs being the fallback in case a runbook does not cover the current failure

what? Amazon definetly has operational people. whatever fancy name they give them.

my point is that you no longer have “classic” sysadmins and a lot of work is already done automatically by machines

Reply on news.ycombinator.com