OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
anandtech.com
anandtech.com
Nvidia makes GSync. So the GPU can control and adjust display refresh rate on the fly. Proprietary, closed source, requires private Nvidia License. AMD makes FreeSync. FreeSync open sourced, added to the HDMI, and Display Port standards.
Now we see Nvidia makes NvLink. Proprietary, very fast, requires private Nvidia license. AMD partners with IBM, Google, etc., etc. to make OpenCAPI an open standard that can revise/replace PCIe3.0
Why does this feel like Microsoft vs Linux but with Hardware Standards.
tl;dr: Nvidia, fuck you.
That's why I was so sad AMD was having trouble later year and hope they pull through. Competition is valuable and benefits the consumer!
I've been guilty of using too many quotes in the recent past. Someone on HN called me on it, and I've since toned it down. Now it's something that sticks out at me.
This is similar. It's also point-to-point, and it has an external story. Skimming the spec, they've even thought about higher latency (200ns) links, and optical. Even with these advantages, I'm unsure how it'll work due to the lack of IP available as compared to PCIe.
Interesting, though. Will be fun to see it play out. It bums me out a little bit though that POWER isn't easily available yet.
The lack of IP is the reason why P8/CAPI stuck to PCIe. In the original design, CAPI was to simply reuse the PCI link layer, not the transaction layer.
With a cache-coherent system based on PCIe, especially when the coherency layer is set at L3 like on POWER8, you are looking at ~500ns latency for a single cache line. This kind of latency is just too much for many applications.
You can configure it in CAUI-10 (10 lanes x 10.3125G) or CAUI-4 (4 lanes x 25.78125G), either way, it's been production-ready for quite some time now. (The docs have numbers, but trust me, you can get full throughput within that 200 ns).
There's even production Agilent off-the-shelf test equipment out there that can fully sample at those speeds (none of that over-sampling tomfoolery, we're talking live, Bill O'Reilly style).
In 1989, UltraSPARC had similar facilities (SBus) to push 100MBit between other Sun machines, so I mean, not too insane comparatively.
* Free with purchase of Virtex® UltraScale™ and Kintex® UltraScale FPGA required haha.
I would like to search the term and understand how they can achieve 200ns latency. Maybe am I the only one who thinks 200ns latency as completion latency?
IMO, I'm a little bit pessimistic for OpenCAPI to reduce this latency much, unless it directly connects inside the chip.
Awesome.
To me it is obvious why Intel is not joining this party.
If this is done properly it means you could make generic motherboards, and generic memory controllers, and all sorts of different accelerators and mix and match them from various vendors. So its no surprise that the smaller players in the market and trying to gang together and the larger player is trying to keep lock-in.
Knights Landing as it stands would already integrate in systems better if it were using an inter-cpu bus than a peripheral bus as well as GPUs, FPGAs, and certainly RDMA/memory window systems like Mellanox.
Inherent distrust of standards aside, this could be a great win for people putting together bespoke systems in interesting configurations (i.e. Google), I don't think there is any downside in theory for Intel except more competition.
Omnipath is used to drive both the Ethernet and PCIe.
I note that Facebook and Intel are missing, which makes me wonder if they are off in a corner somewhere doing their own thing.
Inertia is a very strong decisive factor, especially when you need to make sure that 30+ year-old code still work like it's the case in HPC
Most code that works on Intel works on AMD. It's rare that it doesn't. The HPC vendor will have low risk on migration + get a bunch of competing accelerators at various price points for their problem. This is quite an incentive to move even if there would be stragglers.
The question is : are the DOD/DOE (or the chinese/european equivalent) going to risk millions for their next supercomputers on a new architecture or prefer good old intel ?
Especially when we know that the jump to exascale computation will only be possible if we get rid of the buses and have everything on the same chip (which is one direction taken by Intel).
My point is that the conclusion of the article "Intel has to come up with an answer..." is just plain wrong. This new architecture is an answer to intel new developments. We will have to wait to see what stick to the wall.
(disclaimer: I work for IBM, opinions my own)
In the end, that's also what is at stake, and the reason Intel might not be that interested. They will maybe see the light and see that they would have to steer the collaboration toward a more beneficial direction (for them), but at first, they will probably wait for it to gain momentum before deciding that they have to invest money to push the effort in the right direction (for them).
They have to be a threat for the established company to make a move. What nickpsecurity is saying by telling that they don't need Intel is that they are not yet a sufficient threat and that they are capable of becoming one.
If Intel were involved, then AMD would gain no strategic advantage.
So any HPC application that makes use of abstraction libraries for SIMD and GPGPU code can be easily moved to non-Intel processors.
If those gain widespread adoption, that leave little room for Mellanox's IB platform, hurts NVIDIA's sales of accelerator cards, and cuts AMD's and IBM's processors out of the picture entirely.
nVidia would have loved to have a GPU with a QPI interconnect, but Intel wouldn't let them because they have their own GPU ambitions in Xeon Phi. So they came up with NvLink, which is kind-of PCIe but faster. They don't have any switch asics as yet so they are limited to fully connected topologies. Details on NvLink (without NDA) are scant, but I don't think it has the ability to be multi-node (it really is a bus).
Intel are now making Xeon Phi with on board Omnipath (more like Infiniband), which is curious.
Now waiting for AMD Power9 processors.