Hacker News Reader

I would say 9 times out of 10, you will get better results from a Q4 model that’s a size class larger than a smaller model at Q8. But it’s best not to go below Q4.

View on HN

My understanding is that models are currently undertrained and not very "dense", so Q4 doesn't hurt very much now but it may in future denser models.

That may well be true. I know that earlier models like Llama 1 65B could tolerate more aggressive quantization, which supports that idea.

Reply on news.ycombinator.com