Hacker News Reader

Hacker News

Comment by fragmede | Hacker News Reader

Parent Full thread

fragmede

RDMA is common for high performance applications but it doesn't work over the Internet.

View on HN

> RDMA is common for high performance applications but it doesn't work over the Internet.

RoCEv2 is routable.

* https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet

* https://docs.nvidia.com/networking/display/winofv55053000/ro...

Of course you're going to get horrible latency because of speed-of-light limitations, so the definition of "work" may be weak, but data should be able to be transmitted.

It's a good thing the NIC is connected over pcie then.

You can do GPUdirect over the Internet without RDMA though.

GPUDirect relies on the PeerDirect extensions for RDMA and are thus an extension to the RDMA verbs, not a separate an independent thing that works without RDMA.

Again, you can do what I said. You may be using different terminology, but you can do GPUdirect in dpdk without rdma

You can read/write to GPU buffers with gpudev in DPDK yes. It also uses some of the infrastructure that powers GPUDirect (namely the page pinning and address translation). Because you can use the addressable memory in DPDK buffer steering you can have the NIC DMA to/from the GPU and then have a GPU kernel coordinate with your DPDK application. This will be pretty fast on a good lossless datacentre network but probably pretty awful over the Internet. In the DC though it will be beaten by real GPUDirect on RDMA naturally as you don't need the DPDK coordinator and all tx/rx can be driven by the GPU kernel instead.

This isn't GPUDirect though, that is an actual product.

This is GPUDirect. GPUDirect is the technology that enables any third-party device to talk to a GPU (like a NIC).

> but probably pretty awful over the Internet. In the DC though it will be beaten by real GPUDirect on RDMA naturally

It's being used in many places successfully over the internet. RDMA is fine, but completely breaks the abstraction of services. In many places you do not want to know who is sending or what address to send/receive.

Reply on news.ycombinator.com