Hacker News Reader

Comment by crossroadsguy | Hacker News Reader

I find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it.

View on HN

> I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access

The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.

That is literally my next sentence :D

I still think only privacy is relevant to run on your own infra, never cost.

near.ai tries to solve this

> The advantage is no more 5-hours or weekly session limit anymore

That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive.

API access does not have a 5-hour or weekly limit.

I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.

Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.

If the only variable is money, local models will never make any sense at these prices of infra/subscriptions.

For a lot of people, there are other variables though.

You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.

Than it just makes sense that they already do this.

But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.

Unless you have solar and batteries there aren't cheap energy prices :).

I'm german, i pay $45 US Cents / 40 cents in Euro.

Sooooo :D

Looking ahead, do we think token costs will remain where they are today? Go down? Up?

Your investment in hardware is of course fixed, the token cost amortized over time.

I suppose the other question is, do we think down the road that the models capable of being run locally will improve?

Everything's been trending down so far, so until evidence shows otherwise, I'd assume it will continue that trend. IMO the best time to buy local infra will be when that trend begins to asymptote.

There is so much down pressure, I cannot imagine the token/s price going up. It has been going down and will continue doing so, we are far from a plateau IMHO

Cost/task is probably a better metric to track than raw tokens. Here's Epoch AI's recent report that IMO tracks with my experience (5-10X improvement YoY): https://epoch.ai/publications/the-plunging-price-of-thought

There are many overlapping/compounding curves all happening simultaneously pushing cost down, even on the same hardware - faster software, which includes improvements that massively improve memory/memory bandwidth efficiency for model (especially improved kvcache, specdec/MTP, expert loading becoming standard), improved architectures (no one using full attention, engrams), and a never-ending stream of training improvements.

I think one only has to look at Qwen 3.8 27B to see how top tier a smaller model (<15GB Q4) can be. IMO it's one of the first small models to be able to be run as an agentic coder (it's not quite Opus 4.6-level, but it's not that far off), and with MTP/DFlash2, it's runs surprisingly fast on consumer GPUs. The 30B-class MoEs run even faster, and with the right tuning/post-training I'm sure they can match frontier capabilities in many areas (including coding) over the next months. I think last year the open/local AI picture was a lot more grim than it feels like now, tbt.

Reply on news.ycombinator.com