Hacker News Reader

I would give it significantly more context, if you don't have enough room you can quantize the kv cache with some flags that I don't remember ottomh.

I would go ahead and give it at least 64k but the more the better really!

I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.

View on HN

--cache-type-k q8_0 --cache-type-v q8_0

would drop it from fp16 to 8 bits. You don't want to go below 8 bits if you can possibly avoid it and if you really do need to squeeze just drop the v value to q4_0 but leave the key alone. Unlike model quantisation this really makes things worse quickly.

Reply on news.ycombinator.com