> 8x from somewhere else that I have no idea.
8 GPUs in the box.
8 GPUs in the box.
I'm saying that you are still missing another 8x on the way from 19.5 TFLOPS / GPU to 624 TFLOPS / GPU. 19.5 (base FP32 theoretical peak performance) * 2 (FP16 instead of FP32) * 2 (counting sparse matrix ops as dense ops) * 8 (unknown) = 624 TFLOPS.
x 2 (counting sparse as dense) = 624 tflops
x 8 GPUs = 5 "pflops"
The missing 8x you are looking for is just because tensorcore math is much faster than their normal fma path.