Amazon will rent you infiband-class machines like the C6in.metal (https://instances.vantage.sh/aws/ec2/c6in.metal) with 200Gb/s of bandwidth. With EFA (https://aws.amazon.com/hpc/efa/) you can use HPC features like rDMA.

Heh Heh Heh

A bulk purchase of ~60 FDR IB cards, cabling, and network switches to support them sounds pretty expensive.

That being said, IB FDR gear is "older tech" now so the cards and switches can commonly be found reasonably cheaply on Ebay. The switches tend to be bloody loud though, so they're not something you'd want nearby if you can help it.