Yeah. That makes sense, thank you for clarifying. I updated my original post with a chart from NVIDIA which highlights the H100's capabilities. It doesn't seem unreasonable to expect a 7B model to run at 500 tok/s on that hardware.
Still, the NVIDIA chart shows Llama v2 70B at 750 tok/s, no?