Hacker News Reader

Not if a 96 GB GPU costs $30K?

The OP notes that they can infer a 7B model at 2s/tok using their CPU setup. Equivalent inference on a 4090 ($1600) is up to 400 tok/s, or 0.0025 s/tok. This is an 800x difference.

(to be fair, you can do better than the OP's performance, but we _are_ talking several magnitudes of performance, not to mention energy efficiency, which matters at scale)

> Equivalent inference on a 4090 ($1600) is up to 400 tok/s

What? Who's doing 400 t/s @ 7b on a 4090?

I am, with many optimizations eg dynamic batching, tensorrt, but most important is batch size > 1 -- cheating a little bit in the comparison ;).

batch size 1 is more like 200 tok/s, still a very large improvement. the reason why this is still an interesting comparison is that it's comparing consumer hardware vs scaled datacenter setups. datacenter setups are much more efficient in large part because they are designed to process many inputs at once, fully utilizing the hardware. there are advantages to scale in inference, for the gpu-rich.

Reply on news.ycombinator.com