Hacker News Reader

Comment by pinyen_cheng | Hacker News Reader

I think the future should be efficient quantization which can make model with the same level as Qwen 3.8 27b now accessible from our laptop.

View on HN

This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.

Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week. There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.

So it's not about the model (eg TensorFold/Qwen3.8-27B-MLX-4bit) it's about the inference engine?

The future is already here :)

These 3bit ISTA quants of qwen are supposed to be 95% as accurate as the full model and runs moothly on laptops:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

There are multiple independent tests of this on YouTube.

They aren't "Just as accurate". What you see with these quants is the providers only ever show performance on already saturated benchmarks. You'll see labs put out 1-bit quants Deepseek v4.1 claiming only 1% degradation because they're only showing the benchmarks which tell that story. Then you look at the performance on those same benchmarks for significantly weaker models like Qwen3.8-27B even and it's virtually the same because that benchmark is saturated.

I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.

IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.

This is a really good point that goes the other way as well. Some use cases are great for smaller and cheaper models, but it's not obviously when that's the case so I can bank the savings.

Very much so. Probably a lot of compute is being wasted on using SOTA models for tasks that can be done on way, way weaker models.

All I know is that Strata and these models are the most I have ever gotten out of my machine. I get 70 t/s on a standard gaming rig 9070 XT / 5900X / 64 GB DDR4 @3400.

Yeah, they're surprisingly usable eventually. What I found running the Qwen3.8-Flash-Next IQ3_XXS on CUDA kernel development was, sure it generated faster t/s than 27B dense, but it generated so many more bugs, and when the quantization is lower the reasoning traces get longer. It was more or less always able to figure out what bugs it had made and fix them, but it took quite a lot of time to do. I switched back to Q6 on the dense model and that just went away for the most part. I recently upgraded to 128GB system RAM so that I can up to Q5 on Flash Next. Really excited for Qwen4-Flash when it comes out. Gonna be a real beast.

Not even HLE degrades much when going 4 bit:

https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4?hl=en...

4 bit has always been accepted as the quality floor that retains most of the models capability. Going beyond that brings problems. It doesn't make models totally unusable, but my contention is the quant providers are intentionally not showing an accurate picture, and then everyone is glazing 3-bit quants. They are what they are, but it'd be nice if we had a more accurate shared picture of it rather than just hype and vaguery.

I don't know, it hallucinates quite badly - confuses js with python.

Apparently the ud-q4_k_xl version of qwen 3.8 27b is quite better than the q3 and not much larger. At least per r/localllama. (I'm also trying to find a good local model).

Here's the link btw (also for my own reference):

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qw...

Thank you, will give it a try. To be honest only gemma 4s do well in my re test.

Thanks, I haven't used the gemma models in a while. Last I remembered, their small models weren't bad but obviously showed signs of being "small". (Funnily irl I use Gemini 3.1 pro more than all other models combined).

But is 95% accuracy good enough though? Would it be okay with you if the model got every 20th sõna wrong?

You have to understand that little errors compound very quickly, vibecoding is super iterative.

5% accuracy loss is enormous.

Depending on your application. If you're not pushing the model to its limits, 5% is not very much.

Reply on news.ycombinator.com