This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.

If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.

With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.