Hacker News Reader

Next frontier is achieving good output with that kind of quant dimension

Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?

probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.

haven't tried it yet though.

I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?

They seem to degrade a lot less.

You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.

you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic

This but also the kv quantization shouldn’t be lower than 8 bits.