Amazon is offering instances with 16 Tesla K80 GPUs. I don't know how that is realized under the hood, though.
so GPUs get better bandwidth (P2P etc).
Both have max 32 PCI Express lanes - so you can theoretically hang max 2 PCIx x16 cards off each CPU, but as few of the lanes is reserved for other peripherals, in 4GPU/CPU setup they will practically run at lower speeds, ie 8/8/4/4. However, using system with PCIe root complex [1] can improve the GPU<->GPU communication speed.
[1] https://www.microway.com/hpc-tech-tips/pci-express-root-comp...