Llama.cpp v1 Is Out
huggingface.co
huggingface.co
Sounds like earlier the advice was: for coding, quantizing KV cache is bad, keep it at full F16/BF16.
Sounds like now, 8 bit KV cache is ok for coding, but the question is which one? I see f8_e4m3, q8_0. Would be great to get more info about that.