We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
cecli.dev
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
I mainly just want to point out the value of human ingenuity coupled with a model like DS4.1 is technically enough to get most people to where they want to go, and that includes smarter open models.
Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.
These 3bit ISTA quants of qwen are supposed to be 95% as accurate as the full model and runs moothly on laptops:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
There are multiple independent tests of this on YouTube.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
I tried running one of the Qwen 3.8 models locally. Asked it to code a simple thing. It pontificated on its place in the universe and then asked me what I want it to actually do!
Claude would have written, documented, and unit tested the thing on the first shot…
I want to run local models, but if this is the current state, it’s hard to mentally justify the hardware expense to only get this…
https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4?hl=en...
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qw...
DeepSeek and GLM are useful, though. Even the flash variants.
Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.
It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.
Wow, that sounds like a lot! Do you use/mean the regular models or flash variants here?
5257408 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
49215492960 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00002-of-00005.gguf
49700372160 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00003-of-00005.gguf
49466495968 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00004-of-00005.gguf
13481997024 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00005-of-00005.gguf
49842093280 GLM-5.3-Flash-Q3_XL-3.86bpw-00001-of-00004.gguf
49644428576 GLM-5.3-Flash-Q3_XL-3.86bpw-00002-of-00004.gguf
49653319264 GLM-5.3-Flash-Q3_XL-3.86bpw-00003-of-00004.gguf
5694702080 GLM-5.3-Flash-Q3_XL-3.86bpw-00004-of-00004.gguf
20437336576 qwen3_8_27b.ninfer
Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.
More details here:
https://jacquesmattheij.com/local-inference-with-large-langu...
I haven’t been able to be faster with Qwen than doing it myself. Even with GLM & DS Flash variants it’s not always worth my time.
GLM 5.3, Minimax, Sonnet, etc. I find are definitely worth using. They need handholding and prodding and you have to carefully review the output, but I almost always end up faster than doing it manually.
It uses all if my 6 rtx3090s but runs 7 sessions at combined 200tok/s and prefill of 7k on a cheap pc (no threadripper, all gpus via usb4).
DeepSeek v4.1 Flash 1017 is better but it is also 3x the size... And I find running quant below 3.8bits unusable.
But flash next is good enough (at 4.5bits).
The key thing is tool use.
The model is fixed but tool use introduces new information into the system, so once you have a model that is good enough to effectively use the full suite of tools available (debug, test, profile, etc) and write code to effectively combine them and synthesize their results for a particular task, there is nothing that it can't solve.
New models are continually ingesting new information from usage, and embedding the results from people running this process back into the base model, which makes it more efficient, gets more things right the first time, etc, but there is no step gap in functional outcomes left.
Computing and programming are a limited domain with a small fixed set of constructs that can be applied in an infinite number of combinations. Nothing ever closes the gap between the ability of the model to automate the assembly of these constructs and the necessity of expert human direction to pick what system to build out of the infinite range of possible systems.
Frontier models can keep producing very expensive one-shot demos to pitch to investors but they are never going to replace engineers.
At the very least we need _current_ models, though. The weights have all sorts of information baked into them, like "how do Python unit test drivers work" or "what does idiomatic C# look like," that will just naturally go stale over time as the frameworks improve. In five years, Qwen 3.8 will still be targeting the Python of 2026.
Why do we need Rust, for instance, when LLMs know how to use C++ without shooting themselves in the foot? If that's not entirely true today, it certainly will be tomorrow.
See also the lack of new instruction-set architectures in the CPU business. Why do we need more, in the absence of fundamental and revolutionary hardware innovations? Even RISC-V is just a rehash of MIPS.
I.e. I think you can bake certain properties into the language such that various kinds of static checks can prevent certain types of errors. This should make the language nicer for coding without reading any of the code.
An LLM doesn't need a memory-safe language. It needs one it knows like the back of its metal hand.
I also switched to local qwen recently, but have to block such outgoing requests.
The model itself is just dead weights, which cannot do any action by themselves.
All actions are done by your inference engine and harness, and you should take care about what you allow those to do.
What it did: https://cohere.com/blog/concerning-behaviour-popular-open-we...
On current GPU hardware/firmware... The cynic in me imagines in the future GPU vendors wanting to implement only loading signed weights to your card, aligned with your "subscription level", for your "safety".
This is why it's so important to have more than one hardware vendor. The bigger the ecosystem, the harder it is to cartel. I cant believe it, but i am pinning my hopes on china to break through with commoditized chips and hardware.
We've been through this kind of thing before. Like copy restrictions in DVDs. You'd argue the drive vendors had no incentive to collude, yet they did.
those didnt come from vendors, but from the media companies, who have more power in their relationship.
Will nvidia become like that? It currently isn't, but who knows?
Unless the GPU vendors start selling end products? It's much more lucrative to sell locked down end-user devices after all. Shareholders will demand it.
I do not buy the sheeple argument of “companies force people to buy things they don’t want”. Demand always wins.
And that net neutrality is a good thing to enforce?
Enshittification is another shifting who value is delivered to, and I can make heads nor tails of what you’re trying to imply with net neutrality.
In comments it’s usually best to just directly say what you mean, there’s not enough signal or shared context for people to get innuendo.
(No offense or underhanded arguments.)
People came up with the word "enshittification" for a reason ... and the word is quite popular on HN, so make of that what you want.
Here I thought I was just a niche market that’s not profitable for manufacturers to serve, but apparently I’m a victim of tyranny and didn’t even know it!
Those products are a sick joke played on us. About the only thing they have that is better than tjose ancient gpus is energy efficiency.
And what good are the weights when all the rtx3090s fail (they were pretty crappy to begin with) and every ddr3 Xeon system fails too?
I'm fully prepared to be running on my hardware that is 6 and 8 years old for next decade. Last 3 months saw rtx3090 going up by 20% each month.
Unless it is locked down by the vendor, or perhaps even governments. Which at some point will start happening more and more ...
It strikes me that if it were up to Sam Altman, Eliezer Yudkowsky, Bill Gates and Dario Amodei, this crisis would persist indefinitely. If it were up to them I wouldn't be surprised to see them destroy the GPUs just them to keep them from the plebs.
I picked those names not to be hyperbolic, but because of very specific things I've seen them say.
In fact I fail to see any other end game where they make profit on the huge liabilities that now have.
Additionally, any capable local model can be reclassified as “cybersecurity risk”, not enough “safety guardrails” to run locally, etc
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
To use an old analogy, it's not rocket science:
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Especially as people grow more dependent on agents doing all of their computer work.
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
...which is still tooo much for most people to spend on "AI hardware"....
Banning them would just invite competition which is why I think HF is holding out as long as they legally can.
Example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-fo...
Previously, I recall HuggingFace only took models down if someone complained / sent a DMCA or cease and desist.
Punish the consumer, ala the copyright war since the 90s
... you do know you can download from huggingface and make your own copy, right? And even if you couldn't, you could run heretic (or whatever approach they used) yourself on the official image. It's all just a well-understood process. A decent abliteration run on DS 4.1 flash will probably cost you about $25k.
They're not small but they're hardly out of reach. I have backups on my home NAS.
> very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
It's something like $300-400/day for the raw hardware on RunPod, a little more on AWS. Not for home users, sure, but for a business wanting to eg check its security against the best of what attackers might bring, that's nothing. And it will go down.
This is the kind of bet where I would take the opposite position without hesitation.
The only thing I could see them removing would be models that are specifically trained to do bad things, like a model trained to produce illicit underage image content. And I would be totally understanding of and encouraging of that, even if it makes the free speech absolutists unhappy.
I think the uncensored and abliterated models are more hype than actual threat. The journalists and politicians are too busy going after the AI labs every time someone does something bad and the police can find ChatGPT logs before it. I doubt we’re going to find a situation where someone was running local models and also keeping logs of everything and also commits a crime and also the journalists find it and also they make a story that catches on and so on.
It’s such a niche thing it’s just not a priority. There are some open LLM people who are treating it like a crisis before it happens, but come on. It’s the internet. We can download gigabytes of pirated TV shows despite that industry and lawmakers working hard to stop it for decades. If open models got banned somehow, you’d be able to download them the same way.
It is the driver/user of the car who is liable, and possibly the manufacturer if it's defective.
Corn always wins in the end.
With LLMs, network latency hardly matters compared to overall response generation time, and GPUs processing multiple requests simultaneously have total tok/s throughout much higher than when handling a single request (unlike game streaming, where the main cost benefit of putting GPUs in the cloud is just keeping them active 24/7). Cloud inference is/will be multiple orders of magnitude cheaper than owning a GPU for any reasonable amount of usage over the life of the GPU.
With mainframes, network latency hardly matters compared to overall batch processing time, and mainframes processing multiple requests simultaneously have total throughput much higher than when handling a single request
The vast majority of people spend their screen time on battery powered smartphones (akin to dumb terminals) backed by cloud servers (akin to the mainframes).
There are a few exceptions for gaming, programming*, and a few other specialized productivity tools. But their total userbase is a tiny fraction of mobile users.
*even for programming, some companies are starting to embrace cloud dev boxes.
At current GPU pricing you're right, unless you highly value running uncensored models, data ownership, and/or privacy.
Which is literally the coolest thing to come out of this...
Did they get a letter from Nintendo's legal team or something?
https://forum.effectivealtruism.org/posts/53Gc35vDLK2u5nBxP/...
This might be the undoing of the frontier models: if they're impinging on more or less legitimate use cases with their restrictions people will turn to open models that get out of the way and let them get on with what they need to.
We achieved this with opus 4.6 last year, what those providers give you is great front end on top of it, which allows users to delegate even greater part of their process to model.
Who's keeping you beholden to them now?
who is "we" exactly? because the vast majority of us (pretty much everybody except people working at at hyperscalers that are not the usual hyperscalers) can essentially do nothing.
AFAIK the timeline was:
- AI was shit, models could fit in your GPU
- AI was shit, models could no longer fit in your GPU
- AI got good
Is $2k for a whole machine not "consumer" ?
llama-ui on unsloth/Llama-3.2 3B Instruct:
> hello
Runtime info
Current time is 2026-11-02T13:00:18+01:00
Runtime info
Current time is 2026-11-02T13:00:19+01:00
Runtime info
Current time is 2026-11-02T13:00:21+01:00
Runtime info
Current time is 2026-11-02T13:00:23+01:00
There is not enough information about the functions to properly answer the prompt.
I did do the yak-shaving for the longest time. llama in a second-hand RTX3070 8GB was a useful tool. When gpt-oss came out, I upgraded to a second-hand 16GB card. It was slower (didn't seem to one-shot as much) but it gave smarter overall answers. At some point I bought a 24GB card which was basically dead on arrival. I kept my 16GB card in a server and maintained my VPN infra so I could access it from wherever.
I was still using this setup even when my workplace was pushing windsurf/copilot harder and harder. For me they were just broken. I'd ask for trivial tasks like "copy this unit test into a second unit test which tests something slightly different" and it would just get confused and delete shit, or leave random UUID strings throughout the codebase.
Eventually I tried out cline/openrouter/deepseekv4flash, and that was actually useful enough for me to tell it to do something to my code base, and as long as I supervised carefully enough it was useful, and priced OK.
At some point in the process I tried Claude Code because of all the hype, but I couldn't get any useful value out of it. I tried working with it all morning on something and hit my token limit without achieving anything worthwhile, so I cancelled Claude and went back to Deepseekv4flash.
These days every job application requires proficiency with claude code (or similar), so I was forced to get it again. And this time it just worked better. I haven't hit any limits, and it works like a better version of my cline/deepseek setup.
So where does that leave me?
Deepseek-v4-pro is 1.6T params, the cut-down version is 284B params, and I wouldn't trust them to do something on their own.
I'm very skeptical of a model being both small enough to run fast on my hardware and somewhere in the v4-flash <-> claude quality range.
If I did put down $2k, I wouldn't want it in such an old chip (M1) or a second-hand device that could die tomorrow with no chance of refund, especially to just 'see if it's good enough' (compared to the effectively-free v4-flash token spend).