Local AI is real now. And it's blowing my mind | Hacker News Reader

Local AI is real now. And it's blowing my mind

hive.technology

1 pointtylermhall2 comments

The author's configuration is..... Interesting.

I don't disagree with the title. This spring folks were saying "I use Qwen 3.6 27B (or 35 A3B) as a daily driver". They either had low standards or were lying. The models were good enough technically, they capable enough, that you could see the potential, but the coherence was really not great compared to frontier. They'd lose track of what they were doing, mix things up, etc. Technical ability definitely arrived before coherence.

Then DeepSeek 4 Flash came out and suddenly it was like... Ok. You can run a local model that keeps coherence, and is decent technically. Then we got a rash of models that are both good technically and coherent: DS4 Flash 0731, Qwen 3.8 Flash Next, Qwen 3.8 27B, GLM 5.3 Flash, Hy4, MiMo 2.6 Flash... It just has no stopped. We have definitely crossed the threshold from interesting to very usable.

I like the solar bit. Not sure why the author is using a 4 model setup. My tests with Qwen 3.8 have convinced me it's completely fine general purpose, and dedicated small/vision/etc are useless. I can get 110 TPS single stream on 2x Sparks, with 5M total context, concurrent I think some folks are hitting 200-250 TPS. There's no reason to use anything else right now, it's really remarkable. For an Opus 4.6/4.7 quality model.

I am eagerly awaiting the release of Qwen 4 Flash.

I've not really used it to replace any frontier. More that DeepSeek is something that gets turned on to do useful work when I need it. The smaller models I leave on 24/7 for work that needs very little intelligence, constantly. I run IoT development, so it's clef-flash first, then gemma or qwen 35 A3B to do SOME sort of thought on "does this look weird," relate it to the firmware, and build a report on the fleet. The small ones are fast and cheap on memory; Flash Next takes 99 GB and is half the speed, which is the whole reason for the slots on one box.

My purpose for the stack is not to eliminate frontier. I don't think that's really useful goal to have. My hope is to use it as a "new" place in my stack that I'd never want to pay frontier pricing for. It's private, always on, powered by solar, and I can throw a sort of threshold across my entire stack to have some intelligence everywhere.

Agree that coherence is the change this year. That's what made the title true. I also am looking forward to Qwen 4 flash.

I guess what I'm saying is I now have an "intelligence floor" where I can have an always-on layer that can keep me from only deterministic systems to now deterministic + some amount of (now worth it) intelligence.

I wasn't really willing to do that with subscriptions or API pricing or privacy concerns.

Reply on news.ycombinator.com