5 petaflops in DGX? That alone would put one of those babies into TOP500 top 50, and a superPOD would make no. 1, no? Well, if it could do that performance on Linpack/Rmax.
The 5 "petaflops" number is a creatively constructed marketing number based on FP16 TensorCore "flops", sparse matrix calculations, and then multiplying by 8x for some reason. They basically take the 19.5 FP32 TFLOPS number and multiply it by 32x to get to the claimed 624 "TFLOPS" for a single A100. 8 * 624 = 5 "petaflops". I see they get 2x by actually using FP16 instead of FP32, 2x from counting sparse matrix ops as dense ops, and 8x from somewhere else that I have no idea.
[1] https://devblogs.nvidia.com/nvidia-ampere-architecture-in-de...
[2] https://www.top500.org/resources/frequently-asked-questions/
8 GPUs in the box.
I'm saying that you are still missing another 8x on the way from 19.5 TFLOPS / GPU to 624 TFLOPS / GPU. 19.5 (base FP32 theoretical peak performance) * 2 (FP16 instead of FP32) * 2 (counting sparse matrix ops as dense ops) * 8 (unknown) = 624 TFLOPS.
x 2 (counting sparse as dense) = 624 tflops
x 8 GPUs = 5 "pflops"
The missing 8x you are looking for is just because tensorcore math is much faster than their normal fma path.
Broadcom https://www.broadcom.com/products/pcie-switches-bridges/expr... and Microchip https://www.microchip.com/wwwproducts/en/PM42100 have 98/100 lane PCIe4 fabric switches to offer.