Running Qwen3.8 at 255 tok/s on a single 5090 | Hacker News Reader

Running Qwen3.8 at 255 tok/s on a single 5090

meshapplied.com

15 pointsIr0n13 comments

I struggle with approaching this as I do with many quants, in that there's not really a good measurement of how lobotomized by quantization they are other than "a bit but it's not too bad". Ternary weights immediately make me think "unusably quantized" but I'm presuming that's not the case if someone's standing by this release without massive caveats.

I struggle with deciding whether to take the speed hit on q8 vs q6; I wish it was more standard to benchmark quants or have some way to indicate how much intelligence you're leaving on the table.

I don't mind waiting twice as long for twice as smart; I don't want to be passing up the opportunity of 4x speed-up with 98% intelligence retained. But I have no way to differentiate which is which with quants in any vaguely objective way.

Frustrating, as a hobbyist.

That is true, the lack of good benchmarks will be with us for a while I think. Ternary is not a good quant algo, but because it's so drastic, it allows people to use enormous models on worse hardware, or just increase throughput speed. In the meanwhile, there's research going into how this quantisation can be executed while correcting for the loss. In this case we're not just quantising but also post training the model, and we can "trade" some capabilities to others (ex. post-train the experts for math but lose on biology or whatever).

With that said, I find nvfp4 to be far superior. I think right now all you can do is try out the models, maybe run your own benchmarks if you have.

we achieved this on stock nvfp4 model which is within 5% of fp16 version

we then apply lossless techniques only

dflash2 as spec drafter and patched ninfer inference engine (down projections for short token lengths and adding tool call related schema to drafter shortlist)

see: https://github.com/meshapplied/goethe/pull/4

benchmark numbers are measured using see bench verified traces at avg input 28k and concurrency 1

What's the prefill speed and time to first token? Did you see any slowdown at concurrency for the prefill or decode?

TTFT is around 180ms, prefill speed is ~2.7k tok/sec. Yes, both prefill and decode slows down as concurrency increases (it's not linear). This 255 tok/s is at conc = 1.

Since I really think every post of this kind should include the quant: this is at NVFP4.

How would this compare speedwise to Bonsai 2 on the same GPU?

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf

It's faster and more accurate. Bonsai was a way to demonstrate how effective ternary quant is, and how much it accelerates throughput. If you try coding with it you'll find that it often makes mistakes with tool calls and other syntax related tasks.

This model has higher retention because of the weight complexity and the quant method we built, and the speed loss is compensated via kernel optimisations and speculative decoding.

Have you actually observed these issues specifically with the newer Bonsai 2 (and not the original Bonsai)?

having used both I think bonsai 2 is far ahead, but it still calls tools too often. This isn't always a bad thing, but it seems to have more tool calls per successful run than other equivalents; like it's in-grained to triple check things.

might be a good characteristic for a quantized model to have ..

That's fast, thanks for the hf weights release

The title doesn't apply to the link.

The website is not as good as it needs to be.

There's no information being shared other than these models exist and I don't want to drill down through 3+ links trying to find what you're saying about them.

Who would run a random script on a random page with no other information given?

I'm not saying this is bad, but they definitely need to explain what the heck is going on to the point where a 15 year old can understand it, and provide logical and clear breadcrumbs to navigate through their pages.

Even a FAQ would help.

thanks for the feedback, solved the linking, and will provide more info on the webpage asap.

Reply on news.ycombinator.com