Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars.

The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a specific or reconfigurable topology. Something like InfiniBand, where you can ensure very consistent, very favorable bounds on your latency and throughput. So think of it as being a "custom installation" or "custom cluster" instead.

Nowadays there really are only a few real choices left in high-performance computing hardware: CPU vs GPU, processor architecture (eg POWER vs x86), and interconnection. Everything else is more or less irrelevant.

I have to admit though, I've always wanted to see what you'd get if you built a Cray-2 with GPU chips (eg Pascal) or Xeon Phi as processing modules. The sheer density of chips Cray managed to cram in that case is impressive.

There is no such thing as a 'large commodity cluster.' You can order a pile of commodity parts and spend the next five years fiddling with it as you watch your metrics fall into the toilet, or you can contract with a systems integrator with domain experience to make sure that you can actually accomplish the work you set out to do. When you're moving an established set of software from a known environment into a new cluster, it helps to have a wide array of technical experts and SLAs in place to ensure that the environment -- here meaning the physical compute infrastructure, storage service, and software installation -- is reliable and maintainable, so that your team can focus on their core mission -- in this case, the modeling software.

"Chaos monkey" works fine when you're dealing with random cloudstuff and you are in a position to duck-punch your production code. When you have a large dataset that you need to process reliably and predictably, the mantra shifts to "fail never", and that requires engineering up front in combination with carefully-planned maintenance.

Full disclosure: I've worked on this specific supercomputer. While I'm not on the admin team in question, I do work with them occasionally and I have a lot of respect for them -- their job is not an easy one.

I work for the company that provided the storage for this. Our product is not a collection of commodity components but instead something very much more like an "appliance" that provides the performance at scale that NOAA needs. In general, one of the trends we see/fight is the "Hey, we can get components ourselves and use Lustre to build what we need" And.... they are spot on right. What you need to consider doing this is the amount of maintaining and administering that comes with this solution. The trade-off we see is that some groups, "Academia" for example, will prefer the cheaper solution because they have Grad Students, Post Docs and Undergrad research folks to help make the people cost free. Others, who just want to buy something that works in their infrastructure without adding the people costs would prefer to buy something ready made that "just works"

Bottom line: If you are using your system to get work done, the trend is to spend the money for reliability and availability. If you want to save money, it comes at a price (but is certainly do-able)

Related: Look at Apple, Google, Facebook, Amazon, Microsoft... they are large enough to build their custom systems themselves. Not everyone operates at this scale. Individuals can buy a cheap PC and build a NAS... or they can buy one from Netgear, Buffalo, QNAP that "jsut works" at a price premium.

I think the comments nearby about high communications bandwidth required by dense systems of equations are true -- basically they distinguish a tuned, integrated system sold by a vendor like Cray from a row of racks with commodity systems.

The point I wanted to add is that ECMWF, generally acknowledged to be the leader in NWP, also uses a pair of large Cray supercomputers (http://www.ecmwf.int/en/computing/our-facilities/supercomput...).

Note that the Cray XC30 uses Xeon processors and a custom interconnect. The old ECMWF system used IBM Power chips.

Just a guess...

high fidelity weather modelling algorithms are highly constrained by node-to-node communication latency. So their supercomputer probably has some networked-direct-memory-access or another non-standard setup tuned toward passing data between adjacent nodes quickly.

The branding is still very valuable. This article makes them sound like common blades, ready for virtualization or containerization: http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_...

I don't do weather, but probably it has a lot to do with inter-node bandwidth.