Hacker News Reader

Comment by armcat | Hacker News Reader

Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.

Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.

[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855

View on HN

I think your leading statement is a bit more authoritative than what the linked paper actually demonstrates. It demonstrates your claimed qualities for specific algorithms, methodologies, and models.

It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).

That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.

[0]: https://huggingface.co/vcruz305/DSV4.1-Flash-1017-SAGE-REXL3...

I recently did some toying around with (very) small language models (<200k). I used QAT from the beginning and was surprised that I got byte identical results from the int8 model vs the float32 one.

If that scales with parameters I don't see why QAT shouldn't be the norm.

(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)

Do you think QAT isn't the norm yet?

I expected it to be the norm since GPT-OSS

And things like GSQ-RCO optimization from ISTA labs, EXL that you linked. Preserving bits for certain (large) domains like agentic coding. Pruning others. It really works and creates functional models.

I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.

There is always quality degradation when quantizing, but some models keep surprisingly good performance at aggressive levels of quantization too. DeepSeek v4.1 flash at 2-bit is still a beast for example. In my own experience I would easily trade off accuracy/intelligence for a vast speedup, especially if your setup simply wouldn't run something at 4-bits.

sometimes yes sometimes no :)

that's why different models exist.

This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.

If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.

With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.

Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.

Really? I didn't know, where does it spec this usually?

I have never seen it neither on the link above nor on HF.

Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw

Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.

What do you mean never seen?

On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.

On HF, if you look at the model metadata, practically all of the rank one tensors (or stuff like the input embedding tensor where it's only multiplied by a one hot, which becomes a O(1) select rank 1 tensor from array practically) normally gets stored as fp32.

Ollama will expose the metadata on models too, showing the quantization for each individual tensor.

It is a native 4-bit model, so you would think a 2-bit roughly equivalent to 4-bit of a native bf16 model.

Reply on news.ycombinator.com