Hacker News Reader

I think for O(N^2) transformer inference you need to cache all the activations.

You only need to cache the key/value pairs. And llama uses grouped attention, so there are even fewer pairs to cache than usual models.