Hacker News Reader

Comment by jmpman | Hacker News Reader

How does IB compare to Cray’s new interconnect? All supercomputing nodes these days appear to be water cooled, high density servers with a number of Nvidia GPUs connected to a low latency interconnect fabric, along with a Lustre/GPFS storage cluster. The interconnect fabric used to be the secret sauce, and IB has commoditized it. Why buy Cray?

View on HN

You're correct in that, from the HW side, the "crown jewels" of Cray is the network ASIC's. Everything else is pretty much commodity.

AFAICS, the main difference between the latest Cray network (slinghshot) and IB are:

- Adaptive non-minimal routing, needed to get good performance on their more scalable network topology. IB has basically static ECMP, which works good for fat trees, which scales as O(log N). But for really large machines topologies like Dragonfly which Cray is using become interesting.

- QoS features to isolate jobs from each other.

- Some kind of compatibility with Ethernet. Not sure exactly how this works, but apparently when the NIC communicates with other compute nodes in the cluster it can use the high-performance mode, but if it communicates with some other system, such as Lustre servers, or management systems, or maybe even, say, pulling a container image from an external system, it looks and behaves like an ethernet NIC with the normal Linux TCP/IP stack running on top.

Cray just won a $600M bid for Oak Ridge National Labs' new exaflop HPC cluster using AMD Epyc/Radeon.

Most IB cards come from Intel and Mellanox (recently purchased by NVidia).

IB is a commodity, sure, but it's becoming increasingly tied into the Intel/NVidia ecosystem, and having the ability to go with another interconnect/processor architecture is a powerful strategic advantage.

As far as I know, Intel no longer does the IB-compatiable Truscale, or whatever it was called finally. It's OPA now.

It’s funny then that Omni-Path originated with Intel’s purchase of the Cray interconnect team back in 2012.

https://newsroom.intel.com/news-releases/intel-acquires-indu...

And the Q-Logic IB business.

Likely it's mostly just a customer base grab. I will say there is more to an interconnect than "is the hardware commodity or not" though, you have to be able to show you are able to come up with ways to use it more efficiently than competitor designs.

I don't think that "all" is true, and I'm not sure what "commoditized" means for IB more than any other HPC interconnect. They're all single-vendor as far as I know (as I don't think you can just buy Oracle IB): IB, OPA, Aries (and Slingshot), TOFU, BXI, and maybe others -- I don't know what the Chinese ones actually are.

Gen-Z interconnect is multi-vendor (including Cray and HPE).

There was a talk about it at LCA: https://www.youtube.com/watch?v=MSCvkU0UKIU

"Why buy Accenture?"

I feel, it is the same type "aspirational" acquisition

Reply on news.ycombinator.com