Yes, but it’s happening. Open models run on local hardware is the least-bad version of this. I’m glad you’re able to avoid it all, but for people who can’t, this is the next best thing.
11,584 karma · joined December 27, 2010
> Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup
It’s not “they” talking about it. I am personally running the machine I described, and ~20t/s is the average of what I get at every point in 128k context for coding tasks. Dense models like qwen3.8-27b slow down at longer contexts, but flash-next does not. It’s slower for prose, but I don’t really do much of that. Yes it’s barely usable speeds - i’ll be adding a better gpu in a day or two when it arrives.
To run 4bit quant and 256k context, yes you’re going to have to spend more than $600. But not $20k. I have friends running systems that cost less than $2k getting >80t/s output and >1500t/s prompt processing, at 256k context, 4bit quant. A 5060ti with 16gb vram is all you really need to add to this config to get into that performance range.
I live somewhere with cheap power, which helps - but we’re talking about maybe 250W peak power consumption (180W for the gpu, the rest for the system). So if you ran that full tilt 24/7 it’d be ~6kwh per day.
Assuming you have very high power costs, that could work out to something crazy like $3/day, $90/month. I have no idea how you imagine getting to $1000/month without running a fleet of machines like this cranking out tokens 100% of the time.
Like, I can't imagine opus is cranking through 5T parameters for every token... right?
By buying up all the hardware, they've driven up the prices of sever parts, and because the margins are so much better on server hardware all the manufacturers have switched their lines over to server hardware - leaving the pc market pretty much screwed for gpus, ram, ssds, and hdds.
> A modchip is a small electronic device used to alter or disable artificial restrictions of computers or entertainment devices.
My best guess guess at the reference.
A quick gut check would be to look at the model release date. If it's older than the training cutoff for whatever you used, that's probably not the best thing you could be running.
All used gear. A dell 5810 or similar (x99 motherboard, 2011-3 socket) - should cost well under $100 with no ram or hd, upgrade the CPU to something with lots of cores (e5-26xx v4) for ~$50 (check ebay if there's none local). Buy at least 64gb of ddr4 registered ecc ram - I’d shoot for 96gb - if you pay more than $50 for 16gb you're over-paying. Workstations like this usually have 6 or 8 ram slots so 616 is a great config. If you go up to 256gb, I think it unlocks running deepseek, but I've not looked into that much.
Then the GPU is the hard part. A $100 1070ti or similar with 8gb is the very least thing you can put in and get usable performance. Barely* usable. 10gb, 12gb or 16gb vram is much better. You can also double up - two of the same make & generation gpu to double the vram. You can mix generations, but you have to be careful as there are incompatible combinations. I know AMD GPUs can work, but I have no experience with them so I have no advice to give on that front except don't mix nvidia gpus with amd gpus.
Pascal generation gpus (like the 1070ti) don’t have much computing power so prompt processing suffers. Newer is better, more VRAM is better. Server GPUs are usually cheaper (e.g. the P100) but require screwing around with 3D printed fan shrouds for cooling, and they have terrible idle power consumption.
If you want to spend more, you can get a lot more capable machine - but then you're off my map. Personally if I had more to spend I would set up this exact machine and then put all the extra money into a better GPU. Unless you have money to burn I would avoid any system that uses ddr5, since the ram cost is a big jump up.
If you use claude for coding large complex apps, I will concede that’s more debatable - it depends on how you work with the tools, i think. For me, they work.
You can rent your very shiny version of the tools you use for a very reasonable price, but to me the bent and rust-stained tools I own myself are worth much more.
Not everyone has as cheap power as I do, so you’re right it’s worth factoring in - but in my case it’s a very small factor
Not just as good. I’ve never said just as good. what i will stand by is good enough. What you can get for $600 is good enough to be useful for building software. Since paying anthropic/openai money to continue their sociopathic mission is a hard no-go for me, i’m just very glad there’s another option that passes “good enough* bar.
Anthropic continues to serve haiku - which suggests people have a use for less-than sota models.
A bugatti is undoubtedly a higher performance car than a honda civic. Yet if you're commuting to work in rush hour traffic, both cars will get you there in the same amount of time - and one is going to cost you a lot less.
The pitch of the frontier labs is that you need a bugatti for everything. I don't buy that.
But let's compare apples to almost-apples! Qwen3.8 flash next is $0.47/M on openrouter, so $2.44 per day, so 246 days to break even. I say almost-apples because on this $600 system we're running an IQ3_S quantization. The hosted model is running... nobody knows what quant. It probably varies bases on the time of day.
> You’re also comparing benchmarks for the full model
I wish there were more concrete ways to judge the difference. I've spent a bunch of time using 3 and 4 bit versions of qwen 27b, and my conclusion was that the lower qaunt made more mistakes - but that both systems were able to recover from those mistakes in a coding context. I'm not sure if that holds across the board, but that's the best answer I have so far.
It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
One of my beliefs, call it ideology if you like, is that api cost is much more representative of real cost of this stuff. Good luck switch to local models the same week everyone else decides to switch because subscription prices went up 5x… it’s going to cost more than $600.
Anyway, like it or not, you are giving $200/month to fund another ideology. One that looks to me like a leopards-eating-faces type of ideology.
So not really work, then?
You’re automating yourself out of a job using capital you don’t own, and you think your position will hold? I am skeptical.
Anyway, if it’s worth more than $200/month definitely don’t spend $600 and expect it’ll replace that. But $2400 ($200/month for one year) gets you much closer to sota than $600 - and whatever the line that stops being a rounding error for your business gets you closer still.
How much is it worth to own your livelihood? To me outsourcing to anthropic or openai for $200/month means you’re handing them the ability to kill your business at any moment without even noticing. That’s worth more to me than 30% points of intelligence.
That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.
If you were able to saturate usage of this system, you could pay it off in 3 days of fable time or 6 days of opus 5.5.
But ok those are more intelligent. Let's go with something that scores similar - sonnet 4.6. 5.2M tokens is $78/day and the system is paid off in 8 days.
I know that configuration works, because I am running that configuration right now.
But ok, let's compare: https://benchlm.ai/compare/gpt-6-astra-vs-qwen3-8-flash-next
Astra's score: 85.8/100
Qwen3.8-flash-next's score: 55.87/100
So there's a ~30 pt difference between them. Let's add 5% more for quantization damage.
My perspective is, yes - that's a huge difference. No - it doesn't matter. Flash-next is above the threshold of useful intelligence. I can ask it to do something and it does it. Sure, it burns some tokens on bad ideas on the way, and sure it can't turn a single paragraph prompt into million dollar business. I don't need or want that - I want something I can ask to refactor a module, and it does it and it doesn't stop until the module compiles and all the tests pass. flash-next does that.
Not today. But when it turns into a $600 brick you plug in next to your router, that's gonna work for most people.
For today, I'm speaking to software developers - who can 100% handle setting up a local AI system - it's only as hard as building a PC.
I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?
To be clear, I am not skeptical that it will work this way - I think it's pretty likely llm usage will be come a split of de-centralized and centralized. But I am not very convinced this is a natural law of technology.