I asked a friend who works in a quant firm and he was like yes it’s true, and it is pretty insane.
I think there’s research Microsoft and Google are doing for RDMA over 100G Ethernet for intra data center communication as well. Pretty neat.
I asked a friend who works in a quant firm and he was like yes it’s true, and it is pretty insane.
I think there’s research Microsoft and Google are doing for RDMA over 100G Ethernet for intra data center communication as well. Pretty neat.
The original idea was to let purpose made hardware be distributed across DC rather than every server having to carry it: video codecs on FPGAs, hardware wire speed crypto/compression, databases and k/v stores exposed over RDMA, and remote block storage on SSDs.
I was invited to the opening ceremony for the DC. When company's bosses were showed sfx infused 3D graphs allegedly representing their AI things running on it, I was unable to restrain myself from ruining the atmosphere by asking how it is running when all servers in the DC were shut down :D
Also, there may be ongoing research, but it isn't theory at all. HPCs, HFTs, and the could providers have been leveraging RDMA for a long time - e.g. Infiniband. Doing it over Ethernet (RRoCE) is relatively new, and it isn't necessarily any big leap that is happens over 100G instead of 40 or 1.
However, an interesting point as network links go to 100G+ (esp. for RDMA) is again on the storage/processing side. E.g. a wireshark capture on a 100G connection? ~12.5 GB/second, near max bandwidth for DDR3, and can fill 64GB of RAM in about 5 seconds at full fire-hose. So again the hot-potato of bottleneck will be passed, at least for maximum sustained performance situations.
Side note, AFAIK RoCE exists mostly due to non-technical arguments, particularly the inertia created by existing familiarity and deployment of Ethernet in data centers. I think Microsoft was the one flexing on a standards-body to push it through. It is somewhat of a kludge as Ethernet wasn't designed with RDMA in mind - no guaranteed predictable latency, frames can and will disappear if switch buffers overflow, etc. So IMO "research" into the topic isn't super profound - akin to studying how your sedan might be heavily modified to go off-roading almost (but not quite) as well as a pick-up truck.
Even now many that have the luxury are just going Infiniband from the get-go if RDMA/latency are the key priorities rather than tacked on later.
I'm paying $60/month for 100Mb, and if the trend of the last two decades continues, we should see another 50x increase in bandwidth per dollar over the next decade.
he seems to have more confidence in his bandwidth provider than I have in mine!