Hacker News Reader

Comment by zkmon | Hacker News Reader

Qwen-3.8-27B - the best thing that happened in AI so far.

This thing can accomplish the same results as the frontier models do, if you are willing to play a role of an higher level architect or decision maker. Guide it a bit here and there like a tech lead and you get everything done.

The money you pay for frontier models is exactly for outsourcing that high level direction work to agents.

View on HN

> Qwen-3.8-27B - the best thing that happened in AI so far.

Qwen3.8-Flash_Next should be in the running for that title as well. It's astonishingly good. I use both of them. If I need more memory on the machine for other things I run 27B, If I can get by with much less RAM I run QFN (takes about 102GB on my Strix Halo box with 262,144 context). QFN runs quite a bit faster on my hardware - about 50t/s. 27B with MTP starts out around 22t/s but degrades over time as the context fills up. Both are great models, but I'd give QFN a bit of an edge (at least for the coding projects I've worked on). But both feel on par with Claude Opus 4.6 - 4.8.

Apparently Qwen 4 might land this week, and it uses the same architecture as 3.8-Flash-Next (so various LLM serving tools should be able to handle it already). I'm hoping Qwen 4 includes some smaller models - the last 4B and 9B from Qwen were for 3.5.

If they actually release the weights.

The head of Qwen just confirmed that 4 and 5 will remain open.

Open or "community license"

Doesn't matter for local use anyway.

I remember seeing somewhere credible that it was going to be Thursday this week (tomorrow?), but don't remember where.

Assume you've seen the Apsara conf announcement / the planned 'tiers'[0]. Do you have a planned use case for a 4b / 9b? Hopefully the overthinking will get better on the smaller models.

[0]: https://www.yottalabs.ai/post/qwen-4-max-vs-plus-vs-flash-vs...

I’m really wishing for a 35b-a3b version. It’s just such a great speed-capability trade off.

> Qwen3.8-Flash_Next should be in the running for that title as well.

Unfortunately it still hallucinates, which limits it's usefulness for real world tasks. :(

Every model hallucinates, if not given the right context and tools.

How do you tailor your context and what tools do you use?

give it information it needs to complete it's task, and not more so it doesn't get confused.

I find they hallucinate even with the right context and tools.

So, while useful for some people, for the things I care about they're presently pretty useless. :(

Which quant of QFN are you running on your Strix Halo?

HF is telling me that IQ4-XS is the largest one likely to fit without much contortion.

That's the one I'm running using a fork of the gufo inference engine as gufo upstream doesn't support that quant.

I'm running a regular Q4, what about that quant format justifies the fork?

Agree, I run QFN on my AI Pro R9700 (32GB) using https://github.com/Niko1221/Strata It's even faster than on a Strix Halo. Typically I get around 90 tokens/sec with a small context, down to 70 tokens/sec when it's near the maximum context usage.

Err strata is just a fork of llama cpp with fancier uis, right?

Strata manages MoE expert offloading completely differently, which for QFN means ludicrous speed.

I run QFN I2 XS on my 3060ti + 48GB RAM for it. Get around 2x token with 200k Context window. Good enough for me.

I switched back to glm-5.3-flash from qwen3.8-flash-next.

qwen was faster in t/s, but feels like glm spends less tokens thinking and seems better at finding issues, to the point where it ends up being faster.

Same experience, the 4bit quantized xlarge flash next runs at around 70t/s across 3 radeon v620s. Still have to retry the MTP version with full resolution.

QFN is obscene. It runs faster than 27B on all my setups, and it always outdoes it in my uses (from coding to translation to vision-related): from somewhat to a lot.

What do you mean “outdoes” — over complicates?

Google: Outdo is a verb meaning to do something better than someone else, or to surpass someone or something in quality, quantity, or performance.

Surpasses. At least in my case, 27B tends to waste more time in the reasoning stage, too, so 27B overcomplicates. QFN just does a better job. Sometimes I eyeball, sometimes it's measurable like translation chrF++ 59.3 vs 67.5.

Gemma4:31b is better. Qwen is really good at speed (especially with megakernels), but if its going in a wrong direction, it will go in the wrong direction hard. Gemma can self correct with its thinking.

The only reason Qwen is more popular is because it can run decently fast on MLX, while Gemma needs a discrete GPU with 24gb of ram to run at usable speeds.

Also, theoretically speaking, any small model can be as good as a big model if all they have to do is summarize text. Back before agents were a thing, we had prototype coders running of local inference models from Mistral for ai dev. The workflow was to generate llm summaries of the codebase, and a summary of changes to be made. Then we would grab search results from the web, feed it to an llm to summarize, and then finally combine all of that context into a final inference run which would generate code, then the codebase would be run automatically, any errors would be llm analyzed, and an llm would generate the next change to be made in the above cycle.

Nowdays, we can do that with pi coding agent and Gemma4:31b. But we still have plenty of custom harnesses for things.

...especially with megakernels...

I don't understand why megakernels haven't become the default choice. The speed advantage is crazy. I made a few changes to the openjet megakernel to run in a dual gpu (older gen, not blackwell) box and doubled my token generation speed on qwen3.8-27B (4-bit though).

I keep debating adding a third matching gpu to see if I can push the speed a bit more and move back to the 8-bit quant, but I'm stuck on the idea of just building a Blackwell gpu system despite the cost factor. Plus a third gpu is going to going to be weird for the tensor split and P2P data syncs and my workstation is old with 3rd gen PCIe etc etc.

Architectures (like Gemma4) have things like sliding window attention that make it pretty impractical to do on a GPU, you would have to do additional things to make it work, and the talent to do that, even with LLM assistance is very small.

Ah, that makes sense. I haven't bothered to look at current low-level NN architecture details. I'm a little surprised that there isn't a bumper crop of relatively well-informed talent to do gpgpu work. Although, when if I think about it, it has always been a little bit of a niche area.

The context window left on my 24GB card with Gemma is to low to do any major task though. Maybe small diffs or a single direct task. But Qwen has enough context for agentic tasks.

32k context on my 3090 runs fine.

32k context is quite limited in agentic context. Qwen 3.8 27b is much better in agentic and coding workflows. Gemma4 31b is much better in general assistant and language focused workflows.

Realistically a 32GB card is what you need to run Gemma4:31b well. Doing it in 24GB leaves almost no room for context / KV cache.

32k context is plenty fine. thats 50 pages of single spaced text

Not if you preserve reasoning in the KV cache, 32k ctx is nothing.

Can you tell a little more about your software setup? Been playing with local AI models but struggle to get anything as easy as ChatGPT or Claude. I have a folder with skills, and I can switch between ChatGPT and Claude, and either will do the same work. I’m struggling to find something that can do that locally

Local models requires investment of about $2500 into hardware, so that you can run models at decent tok/sec. You then need to basically write custom harnesses. All of my stuff runs on a llama.cpp backend on a dedicated inference machine, and a client runs custom code built with OpenAPI library to make inference calls to accomplish things.

> need to basically write custom harnesses

Curious, why/how does that help? Or do you have some niche needs?

The latent space of models, which basically steers the token generation into a particular direction is much coarser with the local models compared to big models. Small models known fundamentals well, but do not have the breadth to realize the particularities of combination of fundamental task given certain criteria.

As such, if you want to replicate big model behaviour, you need to basically do context augumentation. How you do that is up to you for your own needs.

But my question remains: isn't something like pi.dev good enough for this? Or are you using custom harness to describe the scripts/skills/etc that you create yourself?

I am cheerfully using Opencode and pi.dev on a local model without thinking about any of that stuff, just running it straight out of the box and it's working great. I have access to an RTX6000 on a remote machine so using it for 'free' and it's replaced Claude for my personal projects. It gets stuck occasionally and gets things wrong from time to time, but so does everything.

No. All agents out there basically operate on skills, mempry, and tools. There are almost none that do context injection with every prompt. You basically need an external system that modifies every submitted prompt with additional context

Okay, just started to read up an Agentic Context Engineering (ACE) & Co,: really interesting. Thank you for the explanation!

Yes, this is a straightforward extension for Pi to have it continually steer the model at every turn. You could write your own with Pi itself or use an existing extension.

> You then need to basically write custom harnesses.

Why? Most existing harnesses let you point them at whatever API you want. I'm currently have a great time with Juggler.

I find myself having to write a lot more skills for Qwen3.8-27b to effectively use tools it hasn’t seen before, versus virtually no skill-writing needed for Claude anymore. It seems to understand novel tools and when to use them pretty intuitively.

For basic tasks, yea. For more advanced tasks, where the agents running on big llms will do more actions autonomously, the smaller models will sometimes stop execution or go off in wrong directions.

Why use llamacpp instead of something faster like exllama?

One RTX4090 with 24G gives me about 200k KV cache with Q4_K_M quant for typica coding tasks, via opencode. I use Q6_K with 60k cache if the work doesn't require scanning too many files. Q6_K is really good.

Could you shed some more light on the differences when you switch from Q4 to Q6_K? Tbh, I prefer having larger context windows since it looks like qwen just figures it out eventually. Since I noticed that it really wants to fill its context by, e.g., reading many files etc. I'd be open to experiment with smaller context windows and less quantization in my ninfer fork [0] if there's a benefit to it. But sometimes I think it's just more art that science :)

[0] https://github.com/tensorninja/ninfer-4090

I had the same trouble but I also had some success. I did a little write up on what I got to work.

https://joeldare.com/a-local-open-weight-model-builds-its-fi...

I think it can be made to work if you’re willing to tweak the prompts.

But, to be honest, I find my time to be worth more than the $20 I spend on a light weight frontier plan that can do this without the hand holding.

I want local to succeed, however, because of things like Anthropic stopping games based on other games.

Shameless plug here: I'm building a coding agent that kicks smaller models on track again if they misbehave. E.g. I noticed that qwen-3.8-27b often writes an intend but then stops and hands the turn back to the user. This is why I implemented lots of "nudges" and paired with a goal mode (and maybe autopilot) does wonders.

Here are some examples: https://caudra.ai/gallery/#nudges

> Guide it a bit here and there like a tech lead and you get everything done

How do you validate and verify that outcome?

Or does Qwen-3.8-27B do the validation and verification for you as well? If so, how do you ensure that's correct?

You check its work and if it seems to do its job well enough most of the time, you just start to trust it.

Personally Qwen3.8-27b has not proven itself to me for the projects I’ve tried to use it for. It doesn’t seem to have as much encoded knowledge about the native Mac development stack.

This seems to be the inherent trade-off with smaller models. They can be just as "bright" as their larger contemporaries but they necessarily can't encode as much knowledge.

I think over time we'll realize this is the way forward, you want the smartest model with the least amount of embedded knowledge and you provide the knowledge externally or merge it into the weights for the domain you're interested in. Then you don't have to train a ginormous model every 6 months to pull in new information.

I think this is what the n-gram table in Qwen 4 (and 3.8-Flash-Next) is for. We’ll see if it comes down to the 27b variant.

A RAG solution would make time to first token awful.

> A RAG solution would make time to first token awful.

in your specific workflows, where specifically is the time to first token a dealbreaker? what's your typical target and what's a "no go"?

I tend to use LLMs conversationally, often as search-and-summarize but also as a pair designer. I find it very difficult to operate in this mode if prefill takes more than, say, a dozen seconds. And that only gets to the reasoning output; depending of effort level it can be multiple minutes to get a reply in a freshly compacted context.

> You check its work and if it seems to do its job well enough most of the time, you just start to trust it.

I don't think you meant to say this but it feels like you're saying "vibes".

It would be better to use a scoring framework like Harbor or similar?

I don’t really have time to be assigning my LLM homework and designing a grading rubric for it. I judge its output like I judge my peers’ PRs.

> This thing can accomplish the same results as the frontier models do

What frontier models are you using? I'm literally sitting with Qwen 3.8 27B locally and trying my damnest to get it to work at least 80% of the quality Astra et al have, and can't get close to it. It keeps being so verbose, regardless of inference parameters, system prompts, reasoning effort, and keeps going on tangents. I'm running the BF16 weights via vLLM, can't imagine how much worse the experience must be with the quantized weights people run with the average consumer GPU.

Even with tight guiding, as I guess you could say I do more "AI pair-programming" than vibecoding, it just loses the plot to often and waste time, enough that it's clearly not an alternative to today's frontier models.

Unless you are trying to prove some hard math, or writing extremely complex code, for general coding tasks (which represent 90% of code out there) 27B-Q6_K is more than enough. Q4 does it well too, but might need a few more iterations.

For comparison, I also used GLM-5.3, Muse-Spark etc, with no significant improvement for ordinary/BAU coding tasks.

> Unless you are trying to prove some hard math, or writing extremely complex code, for general coding tasks (which represent 90% of code out there) 27B-Q6_K is more than enough. Q4 does it well too, but might need a few more iterations.

99% of the time, I'm not. But if it cannot reasonably write a basic CLI for transforming data without properly following a ~1K token long system prompt to the letter, then waste my time on it?

"Ah yes, I forgot about part X" or "I misunderstood the intent of Y" are so frequent compared to SOTA models, that it's hard to justify relying on it, even though I'm able to run it pretty much 24/7 at home, if it just was intelligent enough to actually make sense to do so.

FWIW that doesn't match my experience with Qwen 3.8 27B. After giving it some instructions on what to do (usually i start with what i want to achieve, followed by how i want things to do, then asking it for comments/questions/suggestions - all three - where it gives a bunch and i adjust/correct some of those until a plan is made and written down to a file that it can reference), it does things just fine and has done stuff more complex than basic CLIs - web apps, desktop GUI apps (i have several GUI utilities i'm using daily made with it), etc.

I did had it forget some stuff over long runs that went through several context compactions and i'd expect a larger model to be able to handle more vague instructions or have wider breadth of knowledge (if you're trying to make Qwen 3.8 27B do stuff outside the mainstream you need to provide it with some reference material to use) but i never had a problem with it not following instructions.

The harness might matter though but i'm using plain Pi (which is a very barebones harness with a very small system message) without any addons.

> Ah yes, I forgot about part X

Would something like LMCache help here?

> What frontier models are you using? I'm literally sitting with Qwen 3.8 27B locally and trying my damnest to get it to work at least 80% of the quality Astra et al have, and can't get close to it.

Local models can't get even close to frontier ones; it's just bragging. There's a reason why models like Astra have 2 orders of magnitude more parameters (and run on specialized clusters).

Compared to frontier models, local ones (like Qwen Flash) are much slower, they have much less room for context, and most importantly (people forget to mention) they produce much buggier code. Ah, Qwen also goes into loops sometimes. I've used them on powerful (consumer) hardware.

Having said, no doubt that Qwen-Flash-Next is considerably better than Qwen-3.8-27b - but on the other hand, it's several times bigger. Deepseek Flash also keeps getting better but bigger.

So I tried this.

Qwen-3.8-27B using qwen-code via ollama on a desktop machine with 64G ram and a GeForce RTX 3060 with 12G vram.

I used the owntone-server codebase downloaded from github.

/init

It is unbelievably slow. Ok, sure.

It has hallucinated a macro "DLOG" from its read_file output(!!!) and so tied itself in knots over it because it can't find the macro that is wholly absent from the codebase. Hours later its still going and shows no prospect of writing anything to QWEN.md

How the hell are my results so very different to what you're claiming? What have I done wrong here?

I haven't got anywhere near planning anything with it, much less have it implement something, at any quality. I mean it's just so bad.

Edit: Asked claude which suggested a Modelfile

   FROM qwen3.8:27b
   PARAMETER num_ctx 16384
   PARAMETER num_thread 12

ollama create qwen38-fast -f Modelfile

I'll give that a go. I wonder if you guys just have a bunch of H100s or something like this.

Update: failed abysmally.

    x[API Error: 500 no user query found in messages] (Press Ctrl+Y to retry)

Which means not enough context using PARAMETER num_ctx 32768

I wonder what hardware you need to use Qwen-3.8-27b locally at all..

Thank you, geopolitical competition.

I feel like Alibaba is stealing from the frontier models, and I’m a bit queasy taking advantage of that. If they aren’t distilling and it’s actually true they have superior efficiency as they claim, then why aren’t they way ahead?

I feel like frontier models are stealing the entire internet and the entire whatever any human wrote in the latest centuries, including everything _I_, being an human on the internet, wrote. So I have a hard time being morally concerned about them being distilled.

That happened to individuals, so it is fine. This crime is happening against a company, that is wrong and bad.

Violent agreement here, especially if the ostensible difference is entirely dependent on reciting the correct legal incantation ( "fair use" vs "violates TOS")

100%! I want it all taken back and made open. Individual corps shouldn't own and control it all. They can cry me a river as the local models continue to get better.

Not like anthropic or open ai, they sure downloaded torrents of everything to train, crawl the web non stop, eat open source developers work, git repos etc, not pay ANY but they are not distilling other models at least....

Don't forget destructively scanning books too.

Tbf that's in the same realm as "data centers use all the water!" (they don't, like at all). Google were stripping book spines to scan books into Google Books long before modern AI was even a thing.

Sure if they're doing that to "one copy of 100" feel free to get mad. But one copy of millions of copies? I'm glad when books get digitised.

It is well known that anthropic stole from Qwen and Deepseek for their chinese language training.

Reply on news.ycombinator.com