I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
So much this. Plus everything else you mentioned. The freedom and control is real.
Now I want a local LLM server :( $25k is just so much money.
Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.
A lot more with Haiku equivalents (Qwen3.8 27B) or gemini. It's not frontier, but not that far away.
You may need a AMD EPYC processor and associated motherboard, don't forget a good PSU at least 1200W (300W processor + 400W by card to support spikes of consumption). On some setup (~15k€) you can use 4 CG to go up to 128G of VRAM.
I'm not using a EPYC processor for now, so I get 170tps on decode but very slow prefill, the bottleneck is the PCIe exchange between the cards, I need to setup P2P (it means what you think it means, for GCs).
My setup supports full context (2*18 bytes so 256K), I didn't setup YaRN to get 1M of context (2*20 bytes), but it's an option with 128GiB of RAM, the visual component is extracted from RAM so a little slower. I'm not using multiple agents but some people does, it's god like with 4 graphics cards, on 2 only, it's relatively quick but limited to single user and up to 3 to 4 concurrents requests only.
It's not frontier, but look AA of QFN, it's good enough. DeepSeek V4.1 can run on my setup, but it's slow. On dual B70, there is some PoCs to make it usable (>60tps). It improves every days.
If you need more information, look for local setup users on the Web. There is a lot of people sharing information about benchmarks and setups. L80 is my reference for now.
Edit: I translated the unity.
octet => byte, https://fr.wiktionary.org/wiki/Gio
Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.
My stack uses a local llamacpp ROCM build, modified by https://github.com/stew675/llama-cpp-rdna-boosts
The build uses my machine's ROCM and I just link it instead of embedded all that in an image, like the published ones do.
Then deploying it via docker compose to my local machine. The models I want to run are stored in a docker volume, so it's easy to switch the model while keeping the same build.
The biggest negative for me is that I don't think it's enough to do parallelization through llamacpp (I'd prefer to work on one thing faster than two things slowly). I haven't gotten vllm working yet. Also, once you have one, you constantly start thinking about a second card...
This is an alignment issue. OpenClaw with a local LLM with an email you setup for it could contact the police.
Sorry, you're claiming local LLMs are somehow deterministic and never show unexpected behavior?
Some censorhip and bias towards certain ideology is baked into those models.
While it is not a problem for programming tasks, I would never use it for certain other tasks.
Self-hosting means being able to use abliterated models where the superficial instructions like not describing Taiwan as independent are burned off.
Censorship and bias are related but not synonymous. I agree that it's harder to remove political bias when the training data includes it. We know many of the frontier models are heavily trained on Reddit, for example, [which likely explains why we see such heavy left wing bias.](https://trakkr.ai/bias/findings)
More citations:
https://www.gsb.stanford.edu/faculty-research/working-papers... (article: https://news.stanford.edu/stories/2025/05/ai-models-llms-cha...)
https://www.tandfonline.com/doi/full/10.1080/29974100.2026.2...
https://ieeexplore.ieee.org/document/10817610
https://www.washingtonpost.com/technology/interactive/2026/0...
https://link.springer.com/article/10.1007/s11127-023-01097-2
I am not saying I would not use any of them at all, but I would avoid doing certain tasks with them. And with the testing I have done with abliterated models in the past, it is not 100% successful and it can still hit the same walls as original model.
>abliterated models where the superficial instructions like not describing Taiwan as independent are burned off
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
In open models you can "uncensor" them: https://arxiv.org/abs/2406.11717
This is all this stuff that dries people to pay for frontier models. Even running modest models requires a few thousand, unless you just want really lean ones. People don’t need frontier but it’s not cheap to effectively run, say, a 16gb Qwen 3.6 or 3.8 in the 30b range. Ram is like $15/gb and GPU’s with 16gb vram are hardly cheap.
Businesses were quite happy to spend $5k on technology for a worker if it makes things better, why wouldn't they today.
This is not a straightforward business expense conversation IMO. Otherwise yes I do agree with you. A company should invest in supporting local models for their team.
The lesson it taught me is that swapping is slow and that if it happens during your software development, you're much less productive. When I finally maxxed out the RAM on my machine, I could run linux + emacs + X11 + bash + g++ all at the same time, making me about 10X more productive. It's an investment and lesson that has helped me build my career over decades.
And what happens when the girl at the bar figures out that you TIMESHARE that Lambo?
Ok taking the :door:...
This sort of casual misogyny is at the root of the problem. Shame on you.
Besides, I regularly see people here in HN spend a LOT more than that on hobbies and depreciating assets. A few years back, people with coastal salaries and tech jobs were tripping over themselves to spend $100k on a certain brand of trendy electric car.
My entire career has been based on purchasing the most hardware I could afford at home and using it near its peak. I've given up other life pleasures (dinners out, nicer car/housing) several times to max out my computational capacity.
Also hardware like this, if chosen carefully, often has a viable lifetime over 5+ years
Also, it's not like you're running any full models on local without shelling out 20k+
Local models can do this and with tool use they can even call emergency services for you on your behalf.
> I don't get to hear that it's not a doctor and I should call 911.
Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.
Unfortunately, it's fairly tough for someone to determine if they are themselves a reasonable person.
A model that only ever says yes is a cognitohazard, IMO. Like being a billionaire without the money to insulate you from the consequences.
Sometimes, the requested information should be accompanied with a note like that.
Yes. I've also heard the cost (death) of people falling into pseudosciency rabbit holes.
In case a future user wants to know what a 2026 model would refuse to do, I needed to rotate dozens of passwords last night both on the server and on the clients. When it was time to verify that the rotated credentials worked, I asked for a script that would rate limit the credential checks. Claude/opus5 (opus!!) refused and stopped in its tracks.
In another instance, Claude was making great headway bringing a legacy Windows app up in WINE until it’s saw “auth” somewhere and shut down. In the shutdown state, Claude asked me to reword the prompt to see if I could get it to act on less dangerous instructions. But when I reworded my request, narrowed the scope to be clearly unthreatening, or even redirected the model, it’s still rejected me. I suspect Anthropic uses the less suspicious rephrasing as training material to increase their rejection rate.
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.
Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.
It's not economical today to run my own hardware, but the insurance is worth it to me in the sense that if my bet is wrong, I'm still okay with paying extra for the protection.
(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)
At API pricing (i.e. what all enterprise customers are paying), inference is profitable, even if they're paying usurious rates on cost of capital for capex. Where the frontier labs are getting screwed is that they're in a Red Queen's race with each other, spending tens of billions of dollars per year training the next frontier model, and not being able to recoup the investment on that model before it's obsolete.
That's why you see the CEOs calling for government intervention to "pace the frontier": it lets them reduce investment on training without risking that their competitors keep going and make them obsolete. Unfortunately for them, the Chinese labs exist, and would love nothing more than to take the frontier, so they're in a bit of a pickle.
However, that also means AI will be even more lobotomized since the cloud providers will panic over being liable
People keep repeating this, but it is only remotely close to true if you evaluate inference in a vacuum without any of the other supporting costs to facilitate that capability.
Just a few years ago people were saying the same thing about WeWork: "leasing is profitable, the unit economics of individual locations is net positive", and yet...
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It's hard to go with anyone else unless you're rich.
- they raise the prices 10x
- go bankrupt
- ban you or your country
- decide not to help you with what you want to do for opaque reasons
- you are in jail because the model halucinated something and the company promptly reported you to the police
Better avoid all off these and go local IMHO.
Just note that your local model can 100% do this itself.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It'd stick with it too
It doesn’t have to be black or white…
For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plans… they are cheap for what they offer today.
It's training and tuning new models that cost them money. And that's why they are trying to scare people into thinking AI will kill us all in the next 5 years; they want bans on the open-weight models because they know they won't be able to compete with them.
If that is true, either the American labs are not optimizing correctly or they make a ton of money with inference.
Hydro, wind, solar, nuclear, coal, they are building it all, as we speak, at least 10x more than the US. They are aiming to not just be energy independent, but energy dominant.
You can't now (and certainly won't in the future) be able to run an AI provider in the West as profitably as you can in China.
It's also very clear that providers are running variants of models, especially at the SotA levels. I wouldn't be surprised to find out at some point that 1-2 months after release OAI and Anthropic just run "good enough" quants to save themselves on inference (since they're losing money hand over fist). Another upside of local models is that I don't need to worry about model degradation at any given point.