https://community.fs.com/article/infiniband-vs-ethernet-whic...
https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet
Also... don't underestimate the PCI bus bottlenecks when you put 8x 400GB networking + 8x GPUs. There are ways now to have a tree of PCI switches and avoid overloading the main one, each GPU gets its own networking card and PCI switch.