bitexploder

I have ended up using Argon more and more. If you keep it grounded it is a great model. Extremely thorough. A little slow. It feels like it gets things done better than Opus 5.5. It slowly churns through everything. Not necessarily quality but its adherence to goals seems better and it takes less prompting.

For cyber use cases I am absolutely blown away. Can’t say much more, but it is a serious model.

Just my opinions. I really recommend people take it seriously. For many jobs I just prefer it now. It is a new model so it needs time. Opus 5.5 is a sweet spot as well. I use both extensively.

With 32 GB more of RAM you can run ISTA GSQ RCO IQ3 XXS fast. You can make it work now but it is tight. You can stream experts from disk with builds now. You will get an okay token rate. Try it out if you have some disk space. It is a very good local model. I benched it at that quant. Sonnet 5 class at coding benchmarks.

Get a couple or V100S GPU. Figure out cooling. They will run it /fast/ with the right engine. :)

QFN Q4 K XL Unsloth is basically identical to the bf16 model. It’s KL divergence is very very small. It is competitive with the full model. Also I benchmarked iq3 xxs on Strata for 3 days against deepswe and it scored better than the published benchnarks for the model. If the gsq rco quant prettain tasks fit well it is a great model quant. My data shows it takes 20% more tokens basically than q4 xl. A4 xl is within noise on all my repeated task benchmarks over the public model serving and my own runs of q8. I published the deeswe results. You have to be very specific about model choice and know these models and their quirks.

This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.

https://stealthgoat.com/fnfl/pelicans.html

Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)

You will end up with a sub-par result. The space for small models is more specialized and specific and you should optimize to the specific task and or tasks you want to do. It will get some local model working, but there are a lot of interesting small models in the 4-10B space to look at and play with for your specific tasks. If you expect a full agentic coding agent it's just not going to happen without a larger model like Qwen 35B and ... that is not that great by today's standards, even Muse Glimmer is better.

Dense models are not well suited to this. Depends on your system, but most inference engines let you put the KV cache on the host. They should let you use "Q8" KV cache format as well which takes half the storage of bf16, the default format. You essentially are stuck at having to fit the entire dense model into VRAM because it must compute across the entire model every token. Without it being in VRAM it is slow. MoE models such as Qwen 35B with 3B active parameters are much friendlier to limited VRAM systems. This is why you see systems with 16GB of VRAM running something like DeepSeek Flash if they have a bit of spare system RAM. Usually not at any good token rate, but tolerable.

Dense models also don't like quantization as much and are less tolerant of it. There are some very interesting smaller models like https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4... Dory that I would check out. You need system RAM or to stream a larger MoE model from disk and it won't be fast. You need to specialize the models some to get quality results if you have very limited system resources.

Local AI already is really useful. This has been true since Qwen Flash Next and Qwen 3.8 27B rolled out. The struggle with exactly 64GB of unified RAM is that you end up having to stream the engram table from SSD because you will need almost all of your 64GB to run QFN. A 27B dense model still runs a little slow, even on an M1 Max Mac. However if you have even an older gaming GPU with 12-16GB of RAM and enough system RAM you can run this model (QFN) at a very respectable token rate and intelligence level. It is a Sonnet 5 class model you can run at home. There are new quantization techniques that make these models very good at agentic tasks.

And now if you stretch you can run DeepSeek Flash 4.1.

And if you were willing to roll up your sleeves you could build your own AI lab at home and run this model and models like GLM Flash 5.3 and DeepSeek Flash V4.1 at lower quants. The budget GPUs (like V100) are kind of not budget now, but still <5K to acquire 128GB of fast VRAM that can compete with a 5090 and actually has a lot more interesting options available IMO.

Waiting for 512 GB Mac Studio M5 Ultra still. I guess they are rolling things out now, it has been expected for late October.

I just got GGUF Q4 XL Qwen Flash Next 3.8 working on my V100 engine: https://stealthgoat.com/fnfl/pelicans.html

4 pelicans at once!

A long time ago, in a galaxy far away, I used to write a bunch of ColdFusion with a dude that owned an S2000 :)

It used by be a markup language like:

  <cfif blah="something is true">do this<cfelse>do that</cfif>

But now it has its own scripting language like:

  <cfscript>
  // scripty stuff
  </cfscript>

At least, that is how it used to be :)

I got mine just a few weeks ago for about $750 each. I am a decent programmer and have Opus optimizing the inference engine. I know enough to keep it on track, but GPU programming is something I have literally never done, before, but now I know enough about Volta and how to optimize it to be dangerous.

Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.

And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)

I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.

My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.

You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.

I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.

Not if you have your own TB of fast RAM.

The math. It’s pretty brutal. You have to multiply insanely large matrices in memory. A dense model has to have and multiply across every parameter basically. MoE models work around it, sure, but you still have to hold a lot in memory and density of activated parameters matters. There is not much of a way around it yet. We are potentially getting there, but its not obvious yet.

I have 27B at around 160-170 t/s on my hardware but why bother when QFN is faster and generally better :)

I know I mention it way too often right now, but I am just so impressed with Qwen Flash Next right now. I have been using it locally and it's just such a great little model to work with. It makes mistakes and has its quirks, but in general, it just gets things done and its default manner is just right. It is not overly proactive or making messes for me, even compared to DeepSeek Flash 4.1. I really like DS Flash too, it can just be /really/ annoying in some roles and to chat with some of the time.

I can run Qwen Flash Next on hopes, dreams, DDR4, and ancient data center GPUs at 200 t/s :)

There is nothing inherent in the minimization process that says you must distill from a larger model. It is just the thing we know how to do right now.

No commercial service has 200 t/s prefill though. It is 1000s of pps for prefill. It is largely compute bound and very very fast on modern GPUs and model architectures. That is my point. It is an order of magnitude faster than decode, which is largely bound by memory bandwidth and just has a much harder limit. Prefill tends to look like a small slice in typical compute / token generation budgets. I am not saying it isn't important (I have been working hard on my own inference engine for flash next and am intimately familiar). I am saying it does not dominate. Also compaction and reading the whole prompt tends to be one of the few times you really hit peak pps because it is cached and you can just fill your data pipes for a bit. Also Deepseek showed you can make kv cache really cheap. Decode is still decode.

Models like Qwen 27B 3.8 and Flash Next 3.8 show there is a lot of room to pack more into less parameters.

Artificial analysis has benched it on deepswe

It doesn’t. Decode is by far the most dominant and expensive part of agentic tasks and once prefill hits a certain achievable speed decode dominates the time in ant agentic flows with reasoning. There is a reason input is so much cheaper :)

"Sol 6.1 Medium is producing higher quality output ..." -- When it is actually up and working. Even last night I still received constant capacity messages halting codex over a dozen times and I gave up and shut the workflows down.

I would prefer it work and solve quicker. Opus 5.5 has a reasonable pace to me. Sol does not. I understand efficiency I just don't like using it right now because it has literally hung or failed to complete tasks a dozen times in the last week due to inference capacity.

I have two cards from 2017 running flash next at 200+ t/s on the same model file. Engines are wildly under optimized still. :)

Bench is laughably out sync with reality too. I run things like flash next on an rtx 4080 that it says “won’t run” and its rates are just all over the place. Pure slop.

Are we using the same Thing? Sol 6.1 has run <30 t/s for me for weeks.

They also are not very good lol.

Ehm.. I have a model that runs on my older gaming desktop that scores as well (or better) as this model does on DeepSWE. This model does better than Flash Next in many areas and I am sure is much stronger overall, but hmm.

Gemini Flash 3.8 was heavily optimized for agentic coding. It is not an ignored use case. Flash 3.8 benches extremely well on agentic coding. It was very intentionally made good at coding. So was Argon. It is doing a lot of mission critical porting work for things like Rust. Does Google also build models for things other than coding? Yes. It has a lot of enterprise customers that need good general models too. IDK, I haven't seen anything convincing in Flash 3.8 or Argon that say "Google is ignoring coding" it just took a while to get there since Pro 3.1.

(Disclaimer: I work at teh Goog and spend... a lot of time with these models :)

It is a very efficient model, which I really appreciate. It can definitely feel a little slow. I tend to not hang around for it to do its thing, but for orchestrating and taking somewhat vaguely defined coding goals it is pretty amazing.