Comment by christkv | Hacker News Reader

Top New Best Ask Show Jobs

Comment by christkv | Hacker News Reader

Parent Full thread

Is the limit on the speed on inference a memory bandwidth issue or compute?

"it depends" https://kipp.ly/transformer-inference-arithmetic/

Memory bandwidth/latency, especially when you're at smaller batch sizes.

Depends. One might say its sometimes "cache size limited" too.

Reply on news.ycombinator.com