At which quantization? That's the only thing I want to know.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
haven't tried it yet though.
You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.
> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB
And later
> Standard IQ2_XXS (76.87 GiB, Unpatched)
Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.
Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.