The infiniband stuff is 400Gbps per GPU (3.2Tbps per node).
Each GPU is connected with 400Gbps. The rest ist just the normal dataplane which is independent from the GPUs.
Source: Was personally involved in design of that deployment.
Good to know. I went to recheck, because I swore I didn't see anything about that when I looked earlier, but they now say 3.2Tbps infiniband... not sure if they changed it or I was just blind.