Training Bert, ResNet, Mask R-CNN, and Transformer with 16 GPUs and NVSwitch

Cost: ~$250k. For this price you can build 12 quad GPU or 6 eight GPU machines (RTX 6000). Would not scale quite as well for multi-node jobs, but 48 distributed GPUs are still a lot faster than 16 local GPUs for most scenarios. Not to mention flexibility and convenience 48 GPUs provide for larger research teams.