Preliminary analysis via lm-evaluation-harness + vllm

    model         | disk | wikitext | gsm8k (match/error)
    baseline      | 55G  | 8.00     | 0.50/0.09
    nvfp4-gptq    | 27G  | 8.25     | 0.47/0.9
    nvfp4a16-gptq | 27G  | 8.11     | 0.53/0.9
    bonsai-4bit   | 19G  | 16.75    | 0/0 (eval bug?)

Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show something different, but they are using another eval tool. I'll have to add it to my setup. Also why I'm building a setup instead of taking model devs word for benchmarks. (https://github.com/modelscope/evalscope)

Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)

Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)