With GPUDirect and active-wait kernels, you can get a tight controlled latency and saturate PCIe bandwidth without touching main memory. StorageDirect if you need to write to (or read from) disk.
Depends on the application, batching policy, compute intensity, etc. But you can put 8 NICs and 8 GPUs in one node (and have them communicate through nvlink, so huge intergpu bandwidth!) which I can't for CPUs. You can maybe also get some unobtainium A100X or CX7+H100 to skimp on PCIe if you're well funded...