bitexploder

:shrug: I spend a lot of those token make local models better, resurrecting old hardware to make it more accessible and usable. Frontier models and AI is cool. This is my lived experience doing real systems engineering projects. My real sales manual is: https://stealthgoat.com/deepswe-flash-next.html local models are kind of awesome now and everyone should be using them in addition to frontier models. I will build the stuff I want which includes optimizing local inference engines until they are at my hardware ceiling. I use this capacity in my house to do reversing, security research, and internal OpenClaw type stuff I find useful.

It actually finishes things for you? It's been unusably slow for me. Worse than batch mode anything.

Sol is just hilariously slow. Even Luna is slow. It is unusable compared to Claude. I was pretty unhappy with Opus 5 and really gave OpenAI a shot for personal projects, but their quotas are just too small and their capacity issues in the last few weeks have just been untenable. No good. Opus 5.5 is just a very good model, very efficient with usage, gets things done at a reasonable speed. Sonnet 5.5 is also a very nice worker improvement. It is probably on par with Opus 4.6-4.8.

I have used it as well. It is very dense, but it cites with evidence and has a direct and straightforward approach and manner. It doesn't really have a "tone" or mannerisms the way Opus 5 did. It follows instructions and sticks with pretty clean language by default. Argon is solid and handles long horizon well.

I have a couple of V100 and am working on making Flash Next really fast on them. So far I am at like 193 t/s decode on a specialist 3-bit quant. On shorter prompts. Around 109 t/s on a ggml/llama fork at depth before I gave up and started writing the engine from scratch. I guess I will end at about 220-250 t/s at 128k context size with about 3K prefill on these cards.

https://stealthgoat.com/deepswe-flash-next.html (Total AI pile). I benched this quant for 3 days on DeepSWE in a 16GB VRAM gaming PC. It held 90 t/s on avg I think. It’s solid. The authors claims of minimal intelligence loss hold. Sorry it renders bad in mobile will fix it soon.

It feels great to have a local model ripping at these speeds. I now use Flash NeXT as an active agent managing part of the engine build process and let it do some coding on easier chores :)

I do use DS 4.1 flash a good bit. It's only real fault is that it is a very eager model and I have to system prompt it down to a more reasonable level and it will still take initiative, a little more than I would prefer.

Having Flash Next local at 150 t/s with 250k context is a joy. It’s as good as Sonnet 5. It will spaz out but it was less eager compared to DS Flash 4.1. Both are good but I find I prefer Flash Next. This and Qwen 27B are the models people should be freaking out about.

Flash Next is a basically there. It really depends on what you are doing. This model is great. People forget that they felt Opus 4.6 was a great model and now you have it at home.

Yep... and we already have Opus 4.6 at home (Qwen Flash Next 3.8).

Imagine if they changed the law to reflect what they wanted instead of randomly punishing companies that most likely are doing the same thing everyone else is, but they angered the admin by telling them "no" on some topic.

I have been using these models to make 3d models with build123d — they are great at it. I am never touching Fusion again. Astra is current a bit better than Opus 5.5.

No you.

Microsoft is like $100bn deep into OpenAI.

Real world: I use QFN38 to build 3d models. I use build123d + blender and this model is the first local model that does a good job. It is kind of amazing. This model is as good as DSV 4.1 on most tasks and I think it only loses when specific knowledge matters.

There is a magical quant GSQ-RCO that at 3-bit XS beats the full unquant. I have tested it. It is on par with the full unquant on DeepSWE. Give it a shot, it is magical. ITSA lab, they basically devised a way to selectively quantize parameters. I am a GSQ-RCO truther at this point... it is so fast on my V100s and I can't find an instance where it has acted weird or fell short of the full quant on agentic coding. I Have it 3d model in blender, all sorts of stuff. It's just... good.

You could… actually do that. Same technique as abliteration. Run prompts until it can’t write Java lol. I bet it becomes worse at programming in general though :(

lol. JDQ -- Java Delete Quantization it's the new thing!

These GPUs use about 400W at full tilt (200W each), for reference.

64GB across 2 cards. You need the MoE caching build and enough regular RAM to fit everything else to get decent performance. You should get reasonable performance if you have enough RAM. I have a 4080 that does around 40 t/s right now on a MoE caching build. It's great. Not fast, but chugs along.

Not sure I have a strong opinion but I am sort of okay with current state of affairs. I am optimizing for V100. EOL cards on EOL CUDA. Llama is a good enough base for this. A couple weeks of grunting at Claude has gotten the inference /fast/ for my uses. 150-160 t/s on 2 GPU for 27B and 125 t/s on Flash Next. Asking them to upstream every random feature does not make sense. They sacrifice a lot of speed to maintain stability and a reasonable feature set that works across a diverse range of models and systems. They could maybe merge some features like this and gate them on flags a little faster, but you can cobble together what you need and the big models can figure out how to make it fast.

I am benching Flash next on a 3 bit XXS quant and it is holding just fine against published benchmarks. Using DeepSWE official harness and Pi with absolutely zero benchmaxx or harness config. Install stock Pi and running my agents in it. I am halfway through DeepSWE (it takes FOREVER, even at 125 t/s) and it is neck and neck with Opus 4.7 and Sonnet 5.

On a 3 bit quant btw.

I was skeptical but these results are simply reality now. People have figured out how to selectively quantize the tensors that matter less and shrink these models without losing quality or reasoning. This little Flash Next model just gets things done and is honestly pretty pleasant in terms of its mannerisms :)

It is so surprising to me I don't begrudge people their skepticism but these models from Alibaba represent a fundamental and irreversible shift in what local models can do. Qwen 3.8 27B and Flash Next 3.8 are simply different. But people will catch on. I am doing this on $1500 of data center leftover GPUs (V100)

To add: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC... IQ3_XXS is within a point of the fully unquantized model and IQ3_S actually beats the unquantized model on many tasks! You lose absolutely nothing. It is quantization magic :)

Highly recommend the RCO-GSQ quant by ITSA btw. At IQ3_XSS it is within one point of the fully unquantized model.

Flash next is /really/ close. It is at parity with 4.7 as far as I can tell and basically where Opus 4.8 was. It is a genuinely good model. And I run it at home on $1500 of GPU at 125 t/s :)

That is right now. This model is easily as good as Sonnet5 / Opus 4.7 on DeepSWE. I have benched over half of DeepSWE now on a 3 bit Flash Next quant and it is at parity with Sonnet and Opus 4.6/4.7. It finishes most of the tasks they do. Overall it is within 1 point.

FWIW Qwen 3.8 27B is just slightly behind and basically Sonnet 5 high. I have been benching these models. We have Opus at home. :)

It is also easier than ever to build specs and have nice easy to maintain projects. It just doesn't happen magically via few shot prompts :)

The point of a model like this is aimed at providing local inference. You want the smartest model available, as long as it meets all of your other criteria. That isn't always maximizing on the absolute smartest model. There are many reasons to avoid a frontier model for now for a variety of reasons. All of this is even only relevant in the last 6 months anyhow. So it isn't like this is even some long term trade off or position I am proposing.

I will make sure they get out there soon :)

That is what stood out to me as well. Qwen 3.8 Flash can run on very limited hardware as well and it is at least as good as Sonnet 5 in benchmarks like DeepSWE. With its n-gram design you can get flash next running on very limited GPU resources, as little as 16GB of VRAM.

People follow the latest frontier lab models with great attention and migrate to the next big model on their subscriptions. Meanwhile these local models have quietly gotten REALLY good. It is not even an exaggeration. It has happened in the last couple of months.

"Local model you can run at 40 t/s on a gaming machine that is better than Opus 4.6" is way less exciting than "OpenAI IS DOING CRIME!!! OpenAI SOLVED NAVIER STOKES. DARIO SAYS GLM 5.3 BAD! SLOW DOWN THE FRONTIER!".

(edit: also... totally ignore that 27B dense column over there where Qwen 3.8 27B beats Kolibri on nearly every single benchmark. Why would I choose to run this model?)

Those cards should be able to do a lot more. They support int8 and they have very good processing speed. For reference I have a single V100 running around 900t/s prefill and 100 t/s decode during DeepSWE runs on 27B (4 bit quant). Those cards have half the bandwidth so a realistic decode is going to stop around 50 t/s, but those cards can support NVFP4 more easily than the V100. Combined with a TP build you should be able to get 100 t/s. And you should be able to 2x the prefill of a V100. I have had Claude optimizing llama.cpp for about a week and am at around 300 t/s decode so far on two ancient V100s @ 64 GB RAM.

Also for the 3 people that ever read this and are curious about local models still, Qwen 27B 3.8 matched Sonnet 5 in the 17 DeepSWE tasks I have run so far, solving the exact same 7 it has. Caveat: datacurve combined low/medium/high Sonnet 5 data.