Where do you see 1053 t/s for L2 70b?

From their benchmark table, they achieve 1.8s response time at bs=4 for 256 output tokens, which would be 569 t/s throughput.

At bs=1 you are memory bandwidth bound. So as a simple ceiling we divide the memory bandwidth by the model size (70b is roughly 140gb at fp16). That would yield roughly 24 t/s for H100 and 39 t/s for MI300x.

I'm not really sure how GEMM optimizations alone would yield these sort of improvements at fp16. With quantization, where dequant overheads start to become compute bound, or at very high batch sizes, it makes sense and there has been a lot of work with various kernels in this area. But I'm a bit confused at fp16 and bs=1.

Maybe someone smarter can chime in?

EDIT: Ok I watched their video, which shows 39.75 t/s in the unoptimized (which checks out given the above rough maths) and 313.95 t/s for the optimized. I'm not really sure what's going on here.