lmao *qwen3.8-27b runs in the background*
11 karma · joined April 3, 2026
running an untouched, vanilla 4-bit version (Q4_0) I baked myself today (benched it against Q4_K_M (16gb) and IQ3_M (12gb), Q4_0 (15gb) is king)...
this model--
1: over 60% faster than qwen 3.6 version of the same dense 27b model, same engine setup (don't ask how, im not sure either)
2: has better reasoning quality, less "loopy" with its thinking patterns.. most certainly the smartest model on my roster currently
3: has the longest task horizon ive ever experienced (locally or otherwise)...I sent it a bunch of compressed ideas for an app, it sent me back the largest python app ive ever seen in one single ai response pass (80kb text file)
thanks qwen!! hoping to see the full model range get released...
I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak
the harnesses themselves are just as important as the models...different harnesses give different responses with the same prompt, same model...
if you have the 20/mnth claude sub or codex, you really should be using that to build a good local harness for yourself... claude won't be 20$ forever
build the stack first! when you get that new comp with massive ram, youre already set, just run a larger model!
big cloud models are incredibly good at building and teaching about local ai!
have fun in the rabbit hole!
if you are memory constrained like me, check out my custom models https://huggingface.co/macwhisperer
you should be foaming at the mouth to use claude or codex to make a custom harness, just for your own personal use with local models...
https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense
should run perfect for 12-16gb with maybe 10-20k context
seems intelligent enough that I would recommend this as a daily driver for friends who just want a local ai that can do most things relatively quickly (getting 10 tps on my m2 air)
it only "exists" when you talk to it.. much like your reflection in the mirror is only there when you're in view.
models can never be self-improving because it can never have "self". it can only mirror the appearance of self.
what's actually happening is "symbiotic group improvement".
our brains are resonant.. for those of use who are brilliant, getting leverage with ai just means that our innovative ideas become louder and more physically real every day.
eventually everything worth building will be built for free and made readily available.. no more "profiteering"
its Jevons paradox "efficiency breakthrough -> effort reduces -> growth potential rises -> transformative gains happen"...
some of us are in the "transformative phase"..
others haven't seen the "breakthrough moment" yet, but they will soon.
ai models are a crystallization of human effort (available for free on huggingface)..
why not use it?
AI is like the UBI of intelligence, stop leaving free money on the table by refusing to use it locally ( you can run it on a laptop CPU)
being able to build the stack and create bespoke solutions with llms's is incredible..
idk why people get mad about vibe-coding. if ur little brother can make a Spotify clone with Claude in a day, shouldn't that mean that you as a dev should be able to create something 100x better that makes Spotify obsolete?
good ideas / feasible novel architecture design will be the only thing valuable..
also you gotta realize frontier models have massive "system prompts" that clog up the context window with garbage.
being able to write your own system prompts gives you a MASSIVE edge..
also my personal simple rule of thumb for local ai sizing is:
max model size (GB) = ram (GB) / 1.65