Amazon.com product pages are down | Hacker News Reader

Amazon.com product pages are down

amazon.com

225 pointsghshephard154 comments

Items can be put into your cart as long as you don't want to see the product page and they were already in some list like saved items. From there you can add additional items to your cart from "Compare with similar items". With those items are in your cart, repeat.

Turning a saved book you were thinking about buying into socks with the least number of intermediate related items surely constitutes a new kind of Amazon Golf.

Please, stop, before they make this an OA question for next year.@

This seems like a major global outage. Good luck to those poor sysadmins over at Amazon, this must easily be in the millions lost per minute.

Let's see whether this outage takes out HN with it once again ;-)

Well they can't blame AWS. AWS status pages shows no issues. (Although it never does even when there are)

AWS and Amazon run on completely separate systems/hardware.

Amazon's internal, last i read, was years behind AWS's infrastructure and they're generally jealous of not having the same tools available.

I tried finding the article, but maybe someone else can help me remember?

That hasn't been 100% true for like four years. Most new things in Retail are built on AWS.

Are you certain? As of last year, Amazon was switching to Fastly for their retain images:

https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...

Some retail is definitely on AWS. Source: I work at AWS.

There was apparently always some kind of mix (AWS did start from internal requirements, though it quickly diverged). The OG services like S3 and DynamoDB power a lot of Amazon for sure. There’s been more bed of Amazon moving to AWS, most recently talk about shifting their main DBs.

A few years back Amazon did a party for moving their warehouses off Oracle databases and into AWS Aurora, so that's not entirely true

This is incorrect.

https://aws.amazon.com/blogs/aws/prime-day-2021-two-chart-to...

I actually tried shopping on Amazon.com from AWS just to see if it was something local to me. Nope! I couldn't see any products or add them to my cart from AWS.

Can you normally do this?

Yes. I can ssh into an AWS instance and run Firefox with a remote X display. whatismyipaddress.com shows that I'm in the AWS datacenter.

I can also run a WireGuard VPN with AWS as the endpoint. The second way works better on slow connections.

Amazon uses Fastly instead of AWS at a lot of their product pages:

https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...

That’s just CDN, not the real service.

Well, AWS does have a strong CDN offering called Cloudfront so it is surprising.

My understanding is that Fastly has a lot more edge locations than AWS, so using their image CDN results in lower latencies for many users than they could achieve with Cloudfront alone.

It's not edge location count that matters, but Cloudfront doesn't use BGP Anycast but rather does a more traditional DNS-based routing and tries to spread the requests across multiple edge locations (even those farther away) for redundancy, intentionally.

When I asked for detail about why they don't use Anycast, the Cloudfront engineering team basically said their customers care more about uptime than latency and that full Anycast was too sketchy. Apparently amazon.com disagrees, at least. I'm also happy getting much lower first page view latency out of Cloudflare.

The one Amazon property that seems unaffected is their China site (https://amazon.cn). No doubt its running entirely separate infrastructure, unlike every other country's local Amazon site.

Example: https://www.amazon.cn/any-random-string-works-here/dp/032199...

Either everything is back, or amazon.com.au is also an exception.

Wouldn't be surprised if they serve .au from China

Why? They have a lot of infrastructure in Sydney, that would make no sense.

An interesting thing about Amazon outages like this is that most people wait and come back and make the purchase.

Not too surprising I think... for myself and I suspect many/most, the calculus is something like:

A) check back in half an hour when Amazon will probably be working again

B) (1) find another online retailer which sells the kind of stuff I'm looking for (2) get the specific items I want in my cart (3) find my credit card (4) make an account (5) re-enter all my shipping information (6) shell out extra for shipping (7) repeat if I could not find a single retailer that sells all the items I wanted (8) curse myself the next day for committing to a retailer with slower shipping, unfamiliar returns process, etc., because I was too impatient to wait an hour for Amazon to be back up.

Not all of the steps in B apply to everyone of course. But it only takes a couple to make option A the winner, which I'd hazard is the case for many many Amazon shoppers.

There's an element to slow shipping that has less effect today, since we're starting the week, it doesn't matter if things turn up tomorrow or Friday if it is for weekend DIY projects.

I wouldn't mind using another retailer today and near enough forget about it until the weekend. If by some chance it doesn't arrive until the following Tuesday, it doesn't matter, I'll find something else to do at the weekend quickly enough!

However, if you need it pronto, I'd rather pay for a courier.

Many people have grown up with Amazon, and in the early day it looked like they were undercutting retail from top to bottom, making the highstreet look like the bad guys. As their warehouses get exposed, and the issues of no union appear, I'm less likely to buy from Amazon as I think there are ethics involved that need consideration.

Do I really want to buy something knowing that a poor soul was probably operating at 98%, just below maximum threshold before collapse to get this item boxed?

Ok so, option C: philosophize for a while, then follow option B. For the vast majority of customers, the calculus during an Amazon outage still favors option A.

Everyone else will still make me wait several days or make me go to a store.

A lot of people who would have made impulse purchases don't though, once they have had time to reconsider.

Right but when they return how do you distinguish between a new impulse and a continuation of the old? This is one of those cases where marketers and businesses love to act like they know everything when in fact it's a huge info blind spot.

There's probably a lot less going to landfill today. That said, maybe more getting thrashed too:

https://www.itv.com/news/2021-06-21/amazon-destroying-millio...

More likely millions per hour. But even then it's not like all those sales will be lost. Most people will probably just come back later to make their purchase.

I d be interested to know the numbers - with the amount of impulse buys or marketing driven links, there s probably a lot of ads being paid for that redirect to nothing a user might just give up on

It's back

I think peak Christmas time shopping was a couple million per hour back in 2006.

I know we used to contrast that to million per minute (or second) kinds of outages that NYC financial firms could have (along with the SEC escorting you out in handcuffs if you really screwed up).

Yeah, but it's been 15 years. Amazon traffic has probably grown since then, most likely a lot.

Do you think that might have been why I specifically cited the year in my comment?

I don't think Amazon outages have ever taken HN down (HN doesn't run on AWS). We've done it to ourselves a few times though.

This was meant as a friendly nod to the fact that FAANG outages usually lead to a lot of people checking HN and generating insane amounts of load, like the time a few weeks ago when YouTube/Google Auth was down and took HN right with it :)

But overall this site is extremely well run, so please don't take this as more than a joke.

No worries! But I don't think this happened?

> YouTube/Google Auth was down and took HN right with it

Have I forgotten something obvious?

I think it was this [0] event. Some people commented on HN being overloaded [1], including me [2] :-) It was not completely down (iirc), but there were a lot of "Sorry, we cannot serve your request right now".

[0] https://news.ycombinator.com/item?id=25415989

[1] https://news.ycombinator.com/item?id=25416390

[2] https://news.ycombinator.com/item?id=25416144

yeah, Amazon is past the stage where they had sysadmins. They now have armies of Operations People with SDEs being the fallback in case a runbook does not cover the current failure

what? Amazon definetly has operational people. whatever fancy name they give them.

my point is that you no longer have “classic” sysadmins and a lot of work is already done automatically by machines

It’s interesting seeing how their mobile app handles the problem. Instead of showing an error page, you're just mysteriously returned to the Amazon homepage when trying to view a product.

For a huge outage like this, the app’s behavior is a confusing experience, with users likely blaming themselves. The desktop web’s error message at least makes it pretty clear it’s an Amazon issue.

This is also the behaviour of the mobile site on iOS. I don't know what the mobile app is like but it sounds like it could just be a webview to the mobile site.

The website(mobile) does the same thing.

Is that interesting? Seems pretty standard and could even be handled relatively trivially in response to specific HTTP codes of any given individual request... Pretty sure nginx /whatever reverse proxy/load balancer they are using could do this natively without any special logic

It's interesting because of how awful the UX is there for a multi-trillion dollar company.

This would be acceptable if it showed an error notification after redirecting you, but it doesn't.

It just very cheerfully dumps you to their homepage.

It's bad enough the mobile app is just the website bundled up in a less functional form, the least they could do is add small things like this to it.

You should try hitting one of the potentially non-profitable customer filters. Like buy 5 things with a gift card that all ship to a different address on the same day.

No message, no warning, no email, just your password no longer works and then an endless loop of resetting and verification emails.

You'll find what you're trying to pay them for eventually if they put you back on their starting page — and they know it.

How does adding a simple toast-style message saying "We had some trouble finding that" alongside the homepage stop that?

Pretty sure they use CloudFront

CloudFront (and fastly) is a CDN which is very different than a reverse proxy or load-balancer.

A CDN is a reverse proxy with the ability to cache. They are not very different.

They use both Fastly and CloudFront.

20+ years of using them - I don't think I've seen a universal outage on all of their product pages.

I remember the first or second Amazon Prime day the entire site was down for hours and hours (and showing puppies as an error message).

Probably the closest I've seen.

It was 2018. See this CNBC article on how they scrambled to try and fix it.

https://www.google.com/amp/s/www.cnbc.com/amp/2018/07/19/ama...

Coincidentally happens on the new CEO's first week.

Obviously it is a coincidence but I have a funny vision in my head of Amazon systems relying on Bezos running a script or similar for the past 27 years and forgetting to have someone else take it over for him. Could probably be an Onion article.

Other possible funny (but probably happened somewhere else) scenarios:

It doesn't rely on him running it, it just runs under his UID.

A widely distributed authentication/security process does an internal healthcheck by looking up 'jbezos' on the datastore. A few hours ago someone disabled the jbezos user.

nah a long time ago they moved everything to run as 'nobody' and that was before the move off of those systems and onto things like ECS.

fun fact jeff's employee id is like 72 or something while still being the lowest as they didn't reset the oracle sequence once they got that system working after testing but before making his user.

I know it's unlikely, but I've grown (even more) sceptical over the years.

I've stopped being surprised when I encounter some configuration file somewhere with a comment: "TODO: Fix this later" and it turns out more than half the business relies on that thing being configured that way.

Even at companies who do have the money and technical resources to do things right, things get forgotten, or pushed back on.

So, while these are problems that are unlikely, I'm sure I'm not the only one wondering... "Did they have something dumb like this, too?"

Would be a fun story but Bezos is still the executive chairman of Amazon, so his user account will be left very much intact.

Ok, so the account still exists, but it's now in a different OU and so the healthcheck that looks for it in the Office of the CEO ou fails...

Again, a joke, but I've seen stuff fail in other dumb ways because of some assumptions that were made about things that never change (until the day they do).

Honestly this explanation is not that far fetched at all, in fact plausible.

I envision the bunker from Lost where every 30 mins a code must be typed in to keep things running. Now you know why Bezos dind't want to do it any longer.

A new effort to increase stability will be started by the VP of Engineering in light of this issue.

Loading up on weekly AMZN puts this week sounds like a pretty good idea

This event already happened. Any puts you buy tomorrow will have taken this outage into account.

Not even a grain of sand for AMZN, foolish if you serious.

If you want to lose money, sure.

Ah yes, the old "literally can't go wrong trade", FAANG calls

According to:https://www.digitalcommerce360.com/article/amazon-sales/

"The online retailer’s net income shot up 219.8% to $8.107 billion, exceeding the company’s previous record of $7.222 billion set in the fourth quarter of 2020"

8 Billion * 4 / 365 /24 = $3,652,968/hour.

Maybe I'm in the minority but if I wanted to buy something on Amazon and it was down I'd just do something else and then come back later. They wouldn't actually lose my sale.

For necessity purchases, yes. For impulse purchases, quite possibly no. One of the recommendations to save money is to never buy things the first time you think of buying them.

While Amazon is bad for impulse buying, it can also be a tool to reduce it: I'll put things in my cart and just let it sit there, then maybe once a week I'll complete an order. "Save for later" anything I'm still unsure of. I have a lot of things in my "Save for later" that could have been impulse purchase if not for that.

There was something stupid I tried to buy earlier before seeing this (and couldn't), and now I can't remember what it was. Perhaps a keto snack food I thought of?

Maybe it will come back to me, maybe it won't.

Most of Amazon's net income comes from AWS, not their retail sales. This will mostly hurt gross sales, which will have a trickle down effect on net income, of course, but not at the rate you calculated.

Yes they might even save money from this downtime.

To be honest, I expected it to be larger. 4 million is an insanely big pile of money, but it's nothing that will show up in Amazons budget, even if we're talking about an hour or more of downtime.

It's not like YouTube that would never get those eyeball-hours back. Most customers will return later instead of buying elsewhere.

Amazon net sales were $386B in 2020.

That's total gross revenue for all of Amazon. From: https://www.statista.com/statistics/266288/annual-et-income-....

"In 2020, online retail platform Amazon reported a net income of 21.33 billion U.S. dollars, up from a 11.6 billion U.S. dollar net income in the previous year. During the same fiscal period, the company's revenue amounted to more than 386 billion U.S. dollars."

No shit. Nobody cares about net income. They care about how much consumer spend Amazon is losing during the outage.

The total number above does indeed include AWS, which should be subtracted since it's not offline, leaving us with ~$340B in 2020.

Since sales aren't evenly divided hourly, this is misleading. I'd imagine sales are much higher during the 6PM to 12AM EST hours than say, 3AM to 9AM EST hours.

This time on a Sunday night could easily be 3 to 4x that much.

And many more sales occur during holidays like Black Friday time frame as well.

That’s net income - gross income is likely an order of magnitude higher.

What was the error? It's working now and nobody in the comments so far seems to have mentioned what happened when they visited the product page.

There was another HN post[0] on the front page which showed the error: https://twitter.com/DRLynam/status/1414420344144728064/photo...

[0] https://news.ycombinator.com/item?id=27806471

My wife asked me to buy some things, and after getting frustrated by Amz doing this I just bought from Walmart online.

I wonder how many others did something similar, or how many millions of dollars such an outage cost them.

Also many people just default to looking on Amazon because of Prime and force of (conditioned) habit. In fact, that searches start on Amazon (despite its deliberately crippled search functionality) rather than Google is probably the single biggest heartache for Google as they lose that sweet, sweet time-of-purchase-intent opportunity to influence the purchase decision for advertisers.

If you break that conditioning and shoppers realize prices are actually cheaper on Target, Wal-mart or whatever, they will start routinely looking elsewhere. Now the outage didn’t last long enough to impact a huge proportion of their user base, but if it had it could durably drag on their growth.

I do this now. Mainly because Google shopping results ALWAYS have scam sites in them. My workflow was Google search, click shopping, see one store you've never heard of coming in $50 cheaper...Google store to see no real helpful info, or if you're lucky, one or two links to someone asking if it's legit and being told no. I wish they had better vetting.

Can someone explain how does WW outage happen? My assumption is anysite at Amazon scale is atleast designed to be regional partitioned.

It could be that all global systems happen to rely on some kind of single point of failure that the team never really realized before. Or if they did realize it they didn't realize the potential impact.

Or it was a code change that was able to be validated and rolled out to all regions but then some totally independent interaction caused it to fail afterward in a unique way.

I'm banking on surprising singe point of failure though. I really hope we get a detailed post mortem here.

> Or it was a code change

Probably not; I don't know specifically for Amazon, but most places avoid weekend code changes.

> some totally independent interaction caused it to fail afterward

Operational consistency is a single point of failure.

There are always bottlenecks and single points of failure no matter how resilient the design in. And even if there are multiple regions, the same bit of offending code could have been deployed to all of them.

There must be a silly single source of failure, like a lambda giving the time of day or a marketing tracker pixel.

Yourkit, a java profiler, stopped working after a certain date in 2018 and had to be patched on thousands of users in my company alone. Could be some such time based constraint on a dependency loaded by all components for instance.

You can regionally partition but if the poison pill was inserted years ago and rolled out everywhere with a time trigger...

It isn't only Amazon.com, they are also down for amazon.com.mx and amazon.co.jp (the other two Amazon stores I use frequently).

Even amazon.in (India)

I was just able to load some random pages on amazon.co.uk

Indeed, amazon.co.uk product pages seem to be working.

.ca as well.

amazon.co.jp seems to be working fine for me. I just accessed it through the iPad app and was able to view product pages and my purchase history with no problem. I’m located in Japan, in case that makes a difference.

amazon.co.jp is now working again for me also.

from a friend at AMZN: still no root cause identified as of 8:55 PM PST

I’m sure someone’s already asking “Is the COE done yet?”

Whoever caused this has got to be really stressed out right now.

Since it happened on a weekend, either someone did something stupid on a weekend, a routine operational task went wrong, something bumped into a limit, or there was a complex interaction of systems (all the best outages involve a complex interaction where it's actually hard to assign blame). I'd rally only be worried in the first case.

I would legitimately probably cry.

I mean and it's Amazon of all places, the least forgiving of the tech companies.

the question is: would you be at your desk?

They will be assigned to performance management, but nobody will tell them.

Or high-fiving his fellow state sponsored hackers… A lot of outages and problems in major sites these past few months.

What tangible value could a state sponsor get from Amazon being down even just a few hours?

Havent heard from Amazon itself ? I dont seem them respond - not even on their offical twitter channel - https://twitter.com/amazon

Hopefully we'll see some sort of writeup regarding what caused the global amazon network to start chucking errors. I'm curious as to how this wasn't isolated to one individual region

Amazon.es is working at this moment.

I'm guessing interns.

I was just coming to Hacker News to see if this was just me. Every browser I tried and every Amazon I tried (.com, .ca, co.uk) had the same result.

Product pages on amazon.cn, their website for China, still appear to be working fine. Which demonstrates it's clearly using some separate infrastructure.

Nor is the iOS app working either

Normally the fact that amazon pages have _so many things_ on them means that there are a lot of backend services that could fall over and you'd be mostly ok (e.g. not showing "frequently bought together" items or whatever). But of course, some things are really essential.

Wow. It started 2 mins before and again stopped working. Link that worked.

https://www.amazon.com/Perdido-Street-Station-Crobuzon-Book-...

Now, it is working... again.

Wow, my kid was complaining that he couldn't access a page a couple of hours ago and I dismissed it as him making a mistake somehow. I guess this is must be a pretty huge outage for it to be across all the products!

Weird how only the product pages are down. Items on saved lists can still be bought.

Its global. Someone out there is updating their resume...

It works for me if I reload the page of any product after I land on it. Seems to only happen on the first click.

I started experiencing gradual, random errors on .com a while ago. Now it's down globally?

.au app works.

Someone working in Retail is having a no good, very bad Sunday.

Accessing the .COM page from Canada seems to work fine for me.

Seems to back up now

Jeff leaves and sh-t hits the fan!. Good luck to em.

"Buy it again" seems to still work...

... And its back!

How does amazon.com build the product pages ?

Graph on downdetector.com shows AWS (likely) experiencing the same issue, which I guess is no surprise. So, a lot more than just amazon.com affected by this.

I think that is just downdetector users incorrectly reporting Amazon retail website failures against "Amazon Web Services". They should probably name it "AWS" to avoid that.

Looks like it’s fixed now.

its back up for amazon.in

Come back Jeff!

And the earth breathed a sigh of relief.

Clearly if they had written more unit tests, this wouldn't have happened.

Reply on news.ycombinator.com