Six GPUs on One X16 Slot: R9700s and RTX 3090s Behind a PCIe Switch

This was my ~weekend project. It's a PEX88096 switch card in the CPU x16 slot of an AM4 motherboard, running four R9700s and two 3090s at x8 each. A switch was really the only way I could see to hang this many GPUs off a consumer board.

My favorite result: a P2P copy across the four R9700s moved 54.7 GB/s in total, about twice the ~28 GB/s the x16 uplink carries one way, so that traffic never leaves the switch.

The kernel was less cooperative. Current kernels (7.0.6 on this box at the moment) size the switch's bridge window too small for four 32 GB BARs, so two of the R9700s just came up without them. I ended up writing a small module that lays out the BARs before amdgpu loads to get around it.

It was a very fun adventure and I have a few more posts coming later this week about some rabbit holes I fell into along the way.

That PCIE switch card is cheaper than I expected given the current state of things. I suspect the difficulty keeps most people away from doing a setup like this.

Also found it hilarious that you still had a GeForce GT 710 laying around.