Hacker News Reader

Comment by israrkhan | Hacker News Reader

I would like to see if you can build such a monster yourself, and how much would it cost? Did Amazon designed their own motherboards or used some off the shelf boards?

View on HN

The processors they're using appear to be well in excess of $10,000 each. To get 224 physical cores, you'd need to run them in a 8-CPU configuration, so you've got ~$100,000 invested just in the processors alone.

Memory is absurdly expensive right now, and I'd be shocked if 12TB of it cost any less than another $100,000.

You have to sign a multi-year reservation, and with a cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent. However, over the course of the remaining two years of your reservation, they're making a great deal of money on each reservation.

So yea, I imagine someone could build something like this, but such an individual would need to have very deep pockets.

12TB is only about $7000 in chips. I buy them for my own products. There’s a lot of ancillary things you need but raw dimms aren’t anywhere near that experience.

The 128GB DIMMs are nearly $3000 each (https://memory.net/product/m386aak40b40-cwd-samsung-1x-128gb...) and you need 96 of them.

Where can you get 12tb for $7000?

You'll pay more for the highest density chips, but there are back-planes that support more of the lower density parts. I buy the raw chips in reels not DIMMs. The $7k was based on my costs.

ECC DIMMS at the density required for this probably going to run you substantially more.List price for an ECC DDR4 128GB is like close to $10K US each Nobody pays list, but still.. I’m guessing they have something like 48x256gb dimms — that is serious coin.

Can one even buy an 8 socket motherboard off the shelf?

Once you go beyond dual socket boards (and even then, they're not widely available), you're firmly in the territory of custom form factors, and extensibility usually happens via proprietary daughterboards.

At this point it's not really consumer-grade hardware though, so I wouldn't really classify it under "off the shelf".

The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details.

These systems (from SuperMicro or Lenovo, for example) still fit in a standard 19" rack, and still take standard CPUs, standard DIMMs, and standard PCIe cards.

The main limitation is that the complete "barebones" system is indivisible, unlike with most 1S-2S systems, where motherboards can be mixed and matched among a wider variety of chasses. This limitation also applies to high-density offerings like blades and SM's "Twin" line. I have yet to encounter a situation where such a limitation has ever even entered the conversation.

> The "custom" and "proprietary" are largely irrelevant, in that they're merely internal implementation details.

Agreed. I was merely trying to convey that they wouldn't be the normal [E]ATX form factor boards.

The parent was asking for something "off the shelf", and I took this to mean purchasing a motherboard by itself. You could get something like that from SM[0], but I doubt you'd be able to find a compatible chassis for the motherboard from anyone other than SM. At that point it probably makes more sense to just buy the system[1].

[0] http://www.supermicro.com/products/motherboard/Xeon/C600/X10...

[1] http://www.supermicro.com/products/system/2U/2048/SYS-2048U-...

> The parent was asking for something "off the shelf", and I took this to mean purchasing a motherboard by itself.

The parent did ask about a mobo, specifically, but I think it's a stretch that the question was about the component on its own, at least in the context of the broader conversation of feasibility.

I took the "off the shelf" question to be synonymous with "commodity". That is, the opposite of (truly) custom. One doesn't have to be Google or Facebook and design a board layout to get 8 sockets. One can just order it from the same vendor as all the other commodity hardware.

> it probably makes more sense to just buy the system

I'm asserting that this is what every commercial purchaser does, anyway.

http://www.supermicro.com/products/nfo/xeon_mp.cfm

They also have 4 sockets motherboards for a "general public", and I suspect might have something for a not-so-general public.

> cost of $267,512.88 per year (as calculated from another comment in the thread), I assume their profit margins for the first year are nearly non-existent

NB that this was calculated based on the "All Up Front" pricing, so, for that situation, it's safer to say that their profit for the first year is $300k+ [1] and merely non-existent or negative for the second and third years.

[1] Assuming $400k purchase cost as per https://news.ycombinator.com/item?id=18090058

I would suspect they are building their own HW for this, but this is speculation based on the fact that AWS is building DCs for the IC/Gov and as such are working to accommodate where that is heading, and addressing how the ICs/Gov would want control over such HW.

Reply on news.ycombinator.com