That 500 GB/s is the bandwidth between the NV shader cores and the GPU RAM, not between the shader cores and CPU accessible system RAM. You don't want to be transferring between the two pools that often, as those buses are much slower.
And PCIe Gen 3 is limited to 16GB/s, which is the real bottleneck between the CPU and GPU... if you can use all 16 lanes. Most GPGPU set ups with multiple GPUs only are able to use 4 or 8 lanes each, so you then are stuck with multiple 4 or 8GB/s bottlenecks.