Hacker News Reader

Comment by kennywinker | Hacker News Reader

The floor for a functional local AI system keeps getting lower, dispite ram and GPU prices continuing to go up. An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output with Qwen3.8-flash-next:IQ3_S. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant. For less than $800 you have something that holds its own with where the SOTA was less than a year ago.

I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?

View on HN

Because retail models now dominate the price-performance frontier by a huge margin. An open-weight model wins at just 3% of price points (down from 41% just a few months ago).

https://village.dev/blog/closed-ai-models-now-dominate-the-p...

That chart is comparing the cost of buying tokens, not the cost of generating your own. The system I specced out can generate 5.2M tokens per day.

That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.

If you were able to saturate usage of this system, you could pay it off in 3 days of fable time or 6 days of opus 5.5.

But ok those are more intelligent. Let's go with something that scores similar - sonnet 4.6. 5.2M tokens is $78/day and the system is paid off in 8 days.

You’re comparing API rates, but your original question was about subscriptions.

Also, if I wanted to generate Sonnet 4.6-quality tokens, maybe your suggested setup is the better approach (though I still highly doubt it), but I don’t. I want Opus 5.5 / Fable 5.1 level of quality, because the ROI in terms of human-in-the-loop time saved and how much better and faster I can build is worth WAY more than $200 / month to me. That’s basically a rounding error for my business.

… For now, until the machine you keep feeding eats your business.

Anyway, if it’s worth more than $200/month definitely don’t spend $600 and expect it’ll replace that. But $2400 ($200/month for one year) gets you much closer to sota than $600 - and whatever the line that stops being a rounding error for your business gets you closer still.

How much is it worth to own your livelihood? To me outsourcing to anthropic or openai for $200/month means you’re handing them the ability to kill your business at any moment without even noticing. That’s worth more to me than 30% points of intelligence.

I’m getting like $8k per month in API rate value from each $200/month subscription.

You clearly have an ideological issue with these companies, but I don’t share your views or your values. They could cut me off tomorrow but it wouldn’t kill my business. Worst case, I’d have to figure out how to make do with what you’re already choosing to do today.

I absolutely do have an ideological viewpoint. I hope i haven’t come across as trying to hide that.

One of my beliefs, call it ideology if you like, is that api cost is much more representative of real cost of this stuff. Good luck switch to local models the same week everyone else decides to switch because subscription prices went up 5x… it’s going to cost more than $600.

Anyway, like it or not, you are giving $200/month to fund another ideology. One that looks to me like a leopards-eating-faces type of ideology.

There's nothing wrong with having an ideology or having a set of values that the big AI labs are bad for the world and you don't want to support them, etc.

But in this case it seems like it's led your reasoning astray into this perspective where a small open weight model you can run on cheap hardware is just as good as the value you get from paying $200 to Anthropic. The reality is that it's not even close, at least for some people's use cases.

I really doubt I'll ever need to use local models, even in a scenario where the big labs stop doing subscriptions (unlikely), I could just pay API rates for the open weight models through OpenRouter, or just pay for API rates for Sonnet 5.5 or GPT 6.1 Sol, and adjust my workflows a bit. Still worth it.

I don't know, there are a LOT of things I worry about with AI, but what I'm going to do when the big labs drop subscriptions isn't even in my top 10. Probably not my top 20. Just a non-factor.

> on cheap hardware is just as good as the value you get from paying $200 to Anthropic

Not just as good. I’ve never said just as good. what i will stand by is good enough. What you can get for $600 is good enough to be useful for building software. Since paying anthropic/openai money to continue their sociopathic mission is a hard no-go for me, i’m just very glad there’s another option that passes “good enough* bar.

>They could cut me off tomorrow but it wouldn’t kill my business. Worst case, I’d have to figure out how to make do with what you’re already choosing to do today.

That's just the issue of your access to AI in the worst case.

The actually impactful part of the worst case would be what happens to your ability to have a job, put food on the table, or even afford to live in a house.

I have only so much control over any of that.

>That’s basically a rounding error for my business.

As long as you have a business. AI will "Sherlock" many and commodify all.

Perhaps, but there’s nothing you or I can do about one end of that. I can only control how I respond.

I mean, every person who pays them money helps them continue to do this to us. Part of responding is choosing who to give your money to.

Are you considering intelligence per token?

1 qwen output token != 1 opus output token.

Which is exactly why I compared sonnet 4.6 at the end there. They have similar intelligence scores.

Compare it to deepseek flash or glm. Those are far more capable and far cheaper. Or compare it to a hosted version of the same model. The math becomes much harder to justify when you compare apples to apples.

If all you do is generate tokens all day, sure, host your own infra. For people that aren’t maxing out hardware, using someone else’s is a pretty economical option. $20 with deepseek flash gets you a LOT of agentic use, and as the frontier moves forward with new models, so do I.

GLM5.3 costs $1.40/M on openrouter. 5.2M tokens per day is $7.28/day, which makes break-even take 83 days.

But let's compare apples to almost-apples! Qwen3.8 flash next is $0.47/M on openrouter, so $2.44 per day, so 246 days to break even. I say almost-apples because on this $600 system we're running an IQ3_S quantization. The hosted model is running... nobody knows what quant. It probably varies bases on the time of day.

It really depends on caching and your use case.

My deepseek bill for the last 30 days was $2.91 for 354 million tokens for agentic coding. My heaviest usage month was $14.97 for a billion tokens at deepseek. If you are doing turn based agentic coding, 90ish percent of tokens or more are cache hits.

I have another use case for deepseek that does not hit caching at all, and I pay $2 for 12 million tokens every month. I could get that down by choosing slower providers, but speed matters for this use case and saving $.50 doesn’t.

Both those use cases are bursty. I can’t sit and wait for a server to chew through my requests, especially with my non-cache use case. Anything less than 200t/s output and I start looking for a different host.

As far as quants, direct from deepseek is full quant. Good providers tell you the quant. Even the budget ones typically won’t go as low as iq3, though. FP4 seems to be the lower limit of providers that I see.

I would love nothing more than for local models to be up to snuff, but for my purposes, they get blown out of the water on all fronts by api access. Your machine has a payoff measured in multiple years (assuming you can build it for that price: $300 worth of ram is not much unless you go for ddr3, and a $500 gpu is pretty minimal) and would be a really poor fit for what I’m doing. It seems to be a great fit for you, and that’s pretty awesome.

Qwen3.8-Flash-Next is closer to Haiku than Sonnet in my experience.

Yes I know what the benchmarks say. It does not reflect my experience.

You’re also comparing benchmarks for the full model, but then telling people to run IQ3_S. I found the IQ3_S model to be a substantial step down from 4-6 bit quants.

Fair criticisms. If it's comparable with haiku, that's $13/day or 47 days til break-even (again, assuming I have my system generating tokens 24/7 - which I do not).

> You’re also comparing benchmarks for the full model

I wish there were more concrete ways to judge the difference. I've spent a bunch of time using 3 and 4 bit versions of qwen 27b, and my conclusion was that the lower qaunt made more mistakes - but that both systems were able to recover from those mistakes in a coding context. I'm not sure if that holds across the board, but that's the best answer I have so far.

It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.

> If it's comparable with haiku, that's $13/day or 47 days til break-even

With a $20/month subscription you can basically run Haiku nonstop and never have to worry about limits.

Your comparisons continue to make little sense.

> It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.

This myth has been debunked over and over. There are numerous sites that run benchmarks against the providers repeatedly and the results are always within the same margin of error.

> That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.

But you’re not generating Opus or Fable tokens.

And people buying subscription plans aren’t paying API rates.

I have a Claude subscription and I have hardware to run Qwen3.8-Flash-Next a full order of magnitude faster than the theoretical system you specced above.

In no way would I ever consider my local Qwen3.8-Flash-Next setup comparable to even a $20/month Claude subscription. It’s just not.

I do try to use it for bounded tasks from time to time, but using it for actual complicated work generates so many problems later that it’s not worth it.

Often I can have Opus produce a better result in 1/10th the tokens. The Qwen model often gets things wrong that require 10X as many tokens to debug or fix later. You cannot make comparisons like you’re trying to do.

Your analysis is correct if you’re maxing out the $20 Claude subscription and using the best models to do things that need high quality results.

In my opinion, what we actually have to compare against is the $20 subscription compared to what using Claude or ChatGPT is like with the free plan.

If I use Claude free it cuts me off from using it for hours after a very small number of questions being asked, and it can’t be hooked up to a coding system at all (no Claude code CLI or VSCode extension).

So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).

I can send as many queries as I want in any way I want, including custom applications that would have normally been billed at API token rates.

On top of that, all my data is kept private to my server.

(I’m not counting electricity because my server was already on serving all my self-hosted apps, so a significant amount of electricity is only being added to the load when I’m actively asking questions)

> In my opinion what we actually have to compare against is that $20 subscription compared to what using Claude is like when it’s nearly free.

No, we’re not going to compare $500 GPUs to free tiers.

The contorted logic in these threads makes my head hurt.

> So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).

But the $500 gaming GPU in your closet isn’t providing Claude level intelligence or performance. Not trying to brag, but my LLM hardware is an order of magnitude more expensive than that and I hold no illusions that it’s comparable to what I get from Claude.

If you want to compare against something, compare against Qwen tokens purchased through a provider. Compare like for like.

The $20/month Claude sub is on a different level of performance entirely. There is no direct comparison on quality or speed until you’re spending A LOT more on hardware, and even then the closed models still win.

Let me clarify and I hope this helps you understand where I'm coming from.

The free Claude subscription is an insufficient level of service because what I'm looking for is the ability to send as many queries as my heart desires, and to be able to use it with a coding agent. Therefore, I must spend some kind of money. The cheapest flat price plan I can buy from Claude is $20.

That's why I'm comparing against $20 a month. Of course, maybe for this sparse use case it would be better to use API billing instead of a $20 plan, although maybe that means I'm thinking about billing variability.

I am not comparing it to the level of intelligence that Claude offers. I'm comparing spending $500 one time to spending at least $20/month forever. Yes, it's smarter than the model I'm running, but if I'm not paying that money Claude ends my usage mid-conversation and tells me to come back in 5 hours.

The model I'm currently running is Qwen 3.8 Flash Next using Strata (https://github.com/Niko1221/Strata), the VM has 64GB of system RAM assigned (which was previously sitting unused, with deep apologies to Ram Altman). It's definitely not a bad model, and gets the job done for me.

I would generally agree with you that I'm never going to break even against buying tokens from better open weight models on the open market, or maybe even just buying Sonnet 5.5 tokens, but I'm also gaining the data sovereignty aspect, which is worth a non-zero amount of money.

On top of that, the GPU is an asset with resale value, so I'm not losing the money permanently. It has appreciated in value by $150 since I purchased it, and will probably be worth no less than $200-300 in the next 5 years in the worst case.

Do you want to be using Sonnet 4.6 day-to-day though across all of your work?

Doesn't matter if it's free, you'll spend 3x the time correcting its mistakes.

For what I do, that works. In exchange I get privacy, i get to not fund the big labs eating the world, I get to not worry about hitting plan limits, and I don't have to worry about them killing my account if I ask about the wrong thing.

As much as I might agree with your points, is it reasonable to think that the majority of people or even say 25% of people will feel similarly enough to endure a meaningful drop in quality and intelligence in pursuit of those things you mentioned?

I think most people unfortunately don't care about any of those things, or at least they don't care enough as we are about to see by the trillion dollar IPOs of these AI megalabs.

> 25% of people will feel similarly enough to endure a meaningful drop in quality and intelligence in pursuit of those things you mentioned?

Anthropic continues to serve haiku - which suggests people have a use for less-than sota models.

A bugatti is undoubtedly a higher performance car than a honda civic. Yet if you're commuting to work in rush hour traffic, both cars will get you there in the same amount of time - and one is going to cost you a lot less.

The pitch of the frontier labs is that you need a bugatti for everything. I don't buy that.

From those retailers with billions of debt?

Seems more like a final fight against open weight models than a sustainable business model.

Anthropic has so much cash that they earn a billion dollars a year in interest alone.

Seeing comments about the major AI labs breathing their dying breathes because they have so much debt is really fascinating. The ideologies have started to exist in their own little bubble divorced from reality.

debt has to be paid by their customers at some point. Do you really think on top of the raised hardware costs they have a big enough budget to pay? Or would they switch to open weight as long it is good enough. I'm not sure which side here is the ideology. Where is the profit spike from all these AI customers?

What are you talking about? Anthropic doesn’t have debt. They have so much cash that they earn a billion dollars a year in interest.

Customers don’t have to pay off a company’s debt.

It feels like some of these anti-AI rants don’t even care about the facts. They have a story they believe and they’re going to be angry about it no matter what the facts are.

imho those prices are artificially deflated to serve as a loss leader. Either subscription cost goes up or some other model such as advertising fills the gap.

I don't really understand the argument being made here. The best models are better, but they're not that much better. If I'm paying only 20% as much for 20% less performance, I can make up the 20% and have abundant cash or compute left over to spare.

The other thing that matters to me is the tooling. I could use Deepseek Harness with Claude but I'd rather support the same people that brought me DSH.

> imho those prices are artificially deflated to serve as a loss leader.

As a consumer I don’t care. The price I pay is how much it costs.

I’ve been hearing arguments that they could increase the price at any time. For years.

Instead the opposite has happened: A $20/month subscription has become more capable than it was 6 months or a year ago, by a huge margin. I can get so much done within a day of the $20/month subscription that it’s hard to believe it’s so cheap.

I mean no one should expect them to increase the price in the growth phase where money doesn't matter. Post-IPO when growth has a declining rate, they will (hopefully) have some lock in and blast prices as much as the market will bear.

Hardware will get faster. Models will get better.

By the time they reach this point of raising prices, if they ever do, the cost of models with current intelligence levels is going to fall a lot.

So all of these arguments about buying hardware now and switching to open models now to prepare for future price increases are still wrong if they’re argued on the basis of being locked out of cloud intelligence costs.

The cost for a given unit of intelligence is only going to decrease.

That's wishful thinking IMO. Hardware hasn't been improving at a rapid rate and clearly these companies are losing money and burning investor cash at a ridiculous rate. The party has to stop eventually. The price will go up. I expect everyone will have to pay API rates eventually, possibly even higher API rates than the current ones.

> Hardware hasn't been improving at a rapid rate

Yes it has. GPUs and consumer hardware today is much faster than it was 5 years ago and it will be much faster again in 5 years.

> and clearly these companies are losing money and burning investor cash at a ridiculous rate

As every startup should.

The inference is not losing money. I don’t know if you missed it, but it was revealed a while ago that they’re actually making decent money on inference.

Cool story bro

This makes no sense. DS4.1 flash is the cheapest model basically everywhere

Comparing the sort of models that would run on that to frontier models like Opus 5.5 or Astra is silly

Assuming that you need frontier models like Opus or Astra for every use case that you'd throw at an LLM is also silly

For programming, I care about correctness and intelligent/robust design at the expense of all else

I'd rather wait all day for an impeccable implementation of a single feature than get 10 features worth of "okayish" code

You have the harness pick the appropriate model for the appropriate task. Most high level software development projects can be broken down into tasks, many which do not need the smartest model.

Ok, but as parent said, not every use case is programming. If I want a model to extract or guess column's units from CSV header names plus the column's first 5 values, I don't need Opus 5.5, do I?

Astra and 5.5 came out in september, so I didn't compare to them I compared to "where the SOTA was less than a year ago".

But ok, let's compare: https://benchlm.ai/compare/gpt-6-astra-vs-qwen3-8-flash-next

Astra's score: 85.8/100

Qwen3.8-flash-next's score: 55.87/100

So there's a ~30 pt difference between them. Let's add 5% more for quantization damage.

My perspective is, yes - that's a huge difference. No - it doesn't matter. Flash-next is above the threshold of useful intelligence. I can ask it to do something and it does it. Sure, it burns some tokens on bad ideas on the way, and sure it can't turn a single paragraph prompt into million dollar business. I don't need or want that - I want something I can ask to refactor a module, and it does it and it doesn't stop until the module compiles and all the tests pass. flash-next does that.

> I can ask it to do something and it does it.

I run my own local models, and I can tell that yes, one ask and they do... with many more errors, and much much slower, compared to the lowest levels of frontier models.

Sure, if one has no standards at all, local LLMs definitely are "above the threshold of useful intelligence".

I don't need or want that

Putting aside the absurd straw man, you’re touching on an important point: these small open weight models are sufficient for how you like to work, but it’s not remotely good enough for how I like to work. Opus 5.5 is barely sufficient for how I like to work (many parallel work streams, as little of me in the loop as possible, etc).

> as little of me in the loop as possible

So not really work, then?

You’re automating yourself out of a job using capital you don’t own, and you think your position will hold? I am skeptical.

He's automating you out of a job just as well though, whether you keep yourself in the loop or not.

Haha, yes, it’s not work, just like any EM, tech lead, principal engineer, etc, isn’t doing work either. Everyone knows if you’re not scrutinizing each line of code, it’s not really work.

And I work for myself, so no, not automating myself out of a job.

Although I suppose I’m automating the people I won’t hire now out of a future potential job.

Only if people continue to hire you… or pay for what you make. Hope so for you sake (genuinely!)

I hope so too, but not using the best AI will only put me at a massive disadvantage, and won’t change the macro picture at all.

> So not really work, then?

I work like the parent poster does. I am working harder than before the AI tools came around. I am just working at a higher level. Not too dissimilar from someone moving from programming in assembly language to significantly higher level language.

> as little of me in the loop as possible

You may be confused about this. If I am running just one loop at a time, you'd be right. What modern tools allow me to do is think more like a supervisor that has dozens of engineering teams available to him. I always have a long list of problems I need to solve. I used to sit with one and bang on it until it would pass acceptance criteria. Sometimes that took months. Now I just farm out each task to an agent orchestrator, provide it details on what I want done, a bit of design and architecture and acceptance criteria. I then step away from it and move to the next agent orchestrator and repeat the process for the next task. I usually have 6 or 7 large tasks being worked on at the same time. This is currently my personal limit. I can't babysit more than that. But as the models and harnesses get better I suspect I will have to do less babysitting and this process will get more autonomous.

It may sound like this process would remove the need for engineers. In my organization it is doing quite the opposite. As the organization figures out that we can do a lot more, we change our plans to build more. A lot more. Building still requires engineers, more than before, for us. But only those who can think like architects and can manage large teams of AI agents and orchestrators that work 24x7.

Local models at that scale just aren’t comparable to the frontier. I’m a big DeepSeek glazer but even the gap between full DS 1017 and Opus 5.5 is vast. Local models will never be able to reach that level, there’s real limitations at play. SOTA models are getting bigger and bigger.

Local models are useable and practical today. There is very clear diminishing returns from just increasing model size. You say you are a 'deepseek glazer' but the anthropic/openai glazers were saying the local models we have now were literally impossible just a year ago... and it seems the propaganda is still flowing

>Local models at that scale just aren’t comparable to the frontier.

Don't have to be. They just have to be good enough for the tasks you need.

I might humbly suggest that the tasks you need to be competitive as an engineer in 2026 are different than they were in 2024. And they’ll be even more different in 2028.

If you’re using open weight models to code basically the same as you always have been but just a little faster, while your competitors are using frontier models to go 10x faster, you haven’t saved anything.

Now, granted, the real question comes down to whether they’re going 10x faster, and whether that’ll hold up in the long term. But that’s the gamble as I see it. And to me, open weight models aren’t worth the risk. $200 is 0.5% of my little company’s revenue. It would be foolish of me to try and save that money.

> Local models at that scale just aren’t comparable to the frontier.

For how long? Give it another year, and they will be.

It’s a $20-100 a month outlay for an otherwise low expense discipline to access the best models available, without any local dev ops requirement or expense. If access ever becomes a problem, the open models are there. That’s a fairly straightforward calculation for a lot of people.

I just want to own the tools I use to earn my living.

You can rent your very shiny version of the tools you use for a very reasonable price, but to me the bent and rust-stained tools I own myself are worth much more.

This is a great example of why benchmark racing is useless for LLMs.

I use both models extensively and there is no way I would ever agree that Astra and Qwen3.8-FN are that close in the real world.

I agree with the other comments: These comparisons are silly.

You keep saying that you don’t understand why anyone would have a Claude subscription and reciting ideological reasons not to. I’m assuming this means you don’t use their new models much? That’s the only explanation I can think of for how someone could think that Opus 5.5 or even Sonnet 5.5 and Qwen3.8-FN are close.

I use them both (often Qwen3.8-FN as a subagent) and any claims that they’re actually pretty close read as downright delusional to me.

Which is why nobody’s claiming that they are close. Unless you think ~35% difference is a claim of them being close, i guess.

It doesn't matter because a lot of us won't pay for Astra or Fable.

QFN comes in above Sonnet on some benches I've seen. And let's not forget that Qwen 4 Flash is due any day now.

It is weirdly inverted from the previously standard economics where hardware would keep getting cheaper but those efficiency gains would keep at least partially eaten by software.

> I don't see how anyone justifies a claude subscription.

I have hardware to run Qwen3.8 Flash Next (the model you used in your example) much faster and closer to full quality.

I do not understand how anyone thinks it replaces a $20 Claude subscription in performance or quality. At best I some times have Opus give well defined tasks to Qwen3.8, but I have to be very selective about what is handed over. I also use a lot of Opus tokens reviewing and fixing it.

For real work (not fun projects or experiments) I won’t even bring the Qwen models in to play at all. The gulf between Opus and Qwen3.8-Flash-Next is that big.

So I hope everyone takes comments like this with a huge grain of salt and healthy dose of skepticism. If your work has a high tolerance for slop and you can wait a very long time, the local LLMs can do a lot.

However claims about replacing a Claude subscription with Qwen3.8-Flash-Next? The only way this makes sense is if someone is choosing on ideologies like privacy or avoiding big corporations first, with quality a distant second.

I get downvoted every time I say this, but it’s objectively true. The open models that can be run on consumer hardware are far behind Opus, and quantizing them to Q3 takes them even further behind than the benchmarks show.

Also 100 tok/s prompt processing and 20 tok/sec is painfully slow for real work. The slowness is multiplied by how much extra work the less intelligent models have to do to get tasks done.

> I don't see how anyone justifies a claude subscription.

Certainly I see the benefits in terms of privacy and security. But if you're trying to justify local models on economic grounds, you also need to factor in electricity costs.

(Also, people will say that "less than a year ago" represents an enormous capability gap. But people were already ranting and raving about how capable the models are and how cooked the industry is, considerably longer than "less than a year ago"; so.)

Some of the math I see in this thread justifying local LLM hardware reminds me of “hobby math” where someone wants to justify a hobby expenditure and tortures the comparisons until they think they’re winning.

You do local LLM stuff either because you’re strict about privacy or because you want to play with it.

Any argument that you’re going to replace a Claude subscription with a highly quantized model at 20 tokens/sec is delusional thinking.

I think can both agree that if you use your claude subscription to transform an occasional document, muse out a random idea into a single page app, answer questions with research, etc - as I suspect most subscribers do - yes that could absolutely be replaced with a self hosted model.

If you use claude for coding large complex apps, I will concede that’s more debatable - it depends on how you work with the tools, i think. For me, they work.

Sure, but electricity cost is not a lot for me. i was already running a home server, so it’s mostly just the gpu. The gpu i’m currently running is max 75w which works out to about $0.25/day if it’s operating 100% of the time - the gpu I’m planning to upgrade to runs at 180w or about $0.6/day.

Not everyone has as cheap power as I do, so you’re right it’s worth factoring in - but in my case it’s a very small factor

Can you give example of hardware config ?

I thought I was pretty specific? But ok more details!

All used gear. A dell 5810 or similar (x99 motherboard, 2011-3 socket) - should cost well under $100 with no ram or hd, upgrade the CPU to something with lots of cores (e5-26xx v4) for ~$50 (check ebay if there's none local). Buy at least 64gb of ddr4 registered ecc ram - I’d shoot for 96gb - if you pay more than $50 for 16gb you're over-paying. Workstations like this usually have 6 or 8 ram slots so 616 is a great config. If you go up to 256gb, I think it unlocks running deepseek, but I've not looked into that much.

Then the GPU is the hard part. A $100 1070ti or similar with 8gb is the very least thing you can put in and get usable performance. Barely* usable. 10gb, 12gb or 16gb vram is much better. You can also double up - two of the same make & generation gpu to double the vram. You can mix generations, but you have to be careful as there are incompatible combinations. I know AMD GPUs can work, but I have no experience with them so I have no advice to give on that front except don't mix nvidia gpus with amd gpus.

Pascal generation gpus (like the 1070ti) don’t have much computing power so prompt processing suffers. Newer is better, more VRAM is better. Server GPUs are usually cheaper (e.g. the P100) but require screwing around with 3D printed fan shrouds for cooling, and they have terrible idle power consumption.

If you want to spend more, you can get a lot more capable machine - but then you're off my map. Personally if I had more to spend I would set up this exact machine and then put all the extra money into a better GPU. Unless you have money to burn I would avoid any system that uses ddr5, since the ram cost is a big jump up.

I run my own local models and there’s no world where “normies” can do this.

In fact, I just switched from a 96 GB ram laptop to a 32 GB ram laptop specifically so I can have GPU acceleration because 20 toks can’t replace the 6000toks or whatever astra does

> there’s no world where “normies” can do this.

Not today. But when it turns into a $600 brick you plug in next to your router, that's gonna work for most people.

For today, I'm speaking to software developers - who can 100% handle setting up a local AI system - it's only as hard as building a PC.

My local AI helped me fix hardware configuration issues with my PC... it's even easier than building a PC.

> it's only as hard as building a PC

I asked an AI agent to setup local AI inference for me. I gave it my use cases. It downloaded all the required bits, installed and configured them. Benchmarked different settings and models for my hardware and my use cases and then produced a nice report with findings and recommendations.

It is significantly easier than building a PC.

That's awesome. One warning for others following that route - due to training data cutoffs, and how fast things are moving, claude/chatgpt tend to have very out of date info about what models can be run and what hardware / software is required. Depending on how you ask it to solve the problem, they can lead you astray.

A quick gut check would be to look at the model release date. If it's older than the training cutoff for whatever you used, that's probably not the best thing you could be running.

The normie use cases need less performance, so it could be fine. Review this email, presentation slide, make a flyer for the soccer game, what should I do in Barcelona, etc. If you are not running gas town, you don't necessarily need that much.

E5 doesn't support AVX-512 which is quite important

I wasn't describing the best option, I was describing the cheapest option.

I know that configuration works, because I am running that configuration right now.

> I don't see how anyone justifies a claude subscription.

I want more than "functional"

Cancelled my x20 sub after the most recent usage reduction, will probably cancel my openai sub at month end as well... it really makes no sense anymore, they need to increase usage by like 10x to make it worth it.

Of course not. I want to be automated out of a job after paying for expensive hardware to run a model that doesn't work.

The whole AI world is run by sociopaths that want to make human brains economically uncompetitive comparedto computers. If you think running local models will save you, you're either very pessimistic about the future of AI, or you haven't thought it through.

I prefer my setup to handing my money and leverage over me to the sociopaths.

Reply on news.ycombinator.com