Comment by lhl | Hacker News Reader

Comment by lhl | Hacker News Reader

The old DeepSeek V4.1 Flash has been my workhorse model for the past month (after running tests comparing w/ Qwen 3.8 Flash Next, GLM 5.3 Flash, MiMo 2.6 Flash (including the MOPD fix)) and I've run tens of billions of tokens through it and pushed it fairly hard in both private internal benchmarking and in my own workloads to figure out where it's strong/weak.

Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).

I've replaced the previous model with 1017, and while it's early, a few observations:

- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible

- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model

- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.

- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.

> because I'm guaranteed it's not being changed on me

This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance

200+ tok/s on local infra? That's crazy. How much did you pay for your hardware?

This has to be a distilled model? I don't see any route to 200 tokens/s without serious datacenter hardware

you mean quantized? it’s only got 16B active parameters so it can be pretty fast at decode.

sorry, yes, quantised

I find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it.

> I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access

The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.

That is literally my next sentence :D

I still think only privacy is relevant to run on your own infra, never cost.

near.ai tries to solve this

> The advantage is no more 5-hours or weekly session limit anymore

That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive.

API access does not have a 5-hour or weekly limit.

I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.

Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.

You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.

Than it just makes sense that they already do this.

But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.

Unless you have solar and batteries there aren't cheap energy prices :).

I'm german, i pay $45 US Cents / 40 cents in Euro.

Sooooo :D

If the only variable is money, local models will never make any sense at these prices of infra/subscriptions.

For a lot of people, there are other variables though.

Looking ahead, do we think token costs will remain where they are today? Go down? Up?

Your investment in hardware is of course fixed, the token cost amortized over time.

I suppose the other question is, do we think down the road that the models capable of being run locally will improve?

Everything's been trending down so far, so until evidence shows otherwise, I'd assume it will continue that trend. IMO the best time to buy local infra will be when that trend begins to asymptote.

There is so much down pressure, I cannot imagine the token/s price going up. It has been going down and will continue doing so, we are far from a plateau IMHO

Cost/task is probably a better metric to track than raw tokens. Here's Epoch AI's recent report that IMO tracks with my experience (5-10X improvement YoY): https://epoch.ai/publications/the-plunging-price-of-thought

There are many overlapping/compounding curves all happening simultaneously pushing cost down, even on the same hardware - faster software, which includes improvements that massively improve memory/memory bandwidth efficiency for model (especially improved kvcache, specdec/MTP, expert loading becoming standard), improved architectures (no one using full attention, engrams), and a never-ending stream of training improvements.

I think one only has to look at Qwen 3.8 27B to see how top tier a smaller model (<15GB Q4) can be. IMO it's one of the first small models to be able to be run as an agentic coder (it's not quite Opus 4.6-level, but it's not that far off), and with MTP/DFlash2, it's runs surprisingly fast on consumer GPUs. The 30B-class MoEs run even faster, and with the right tuning/post-training I'm sure they can match frontier capabilities in many areas (including coding) over the next months. I think last year the open/local AI picture was a lot more grim than it feels like now, tbt.

talking about "my setup" should require the good-old-days post footer that includes hardware details.

It's part of a node that's used for more serious things (training, RL) and supports multiple users, so HW price doesn't really enter the picture. If you were running locally, DS4.1F has 300GB of tensor weights so you'd ideally want at least 4xPRO 6000 (which I'd guess should get you in the 150 tok/s range w/ DSpark), but people are using Xolotl/Maya/etc w/ quants and getting 25 tok/s decode on a single 4090/3090, so it can be run at a reasonable price.

For best "bang per buck" I think Qwen 3.8 Flash Next (or Qwen 3.8 27B even) makes a lot more sense for most people - you can run qwen38fn at 100+ tok/s w/ Strata (hot expert offloading) 200+ tok/s on 2 x PRO 6000, it does almost as well as ds41f at most tasks (it's much less token efficient than other options, and it sucks at AI/ML tasks though).

CoT = Chain of Thought (I had to look it up)

Thanks for the insights. If you edit your post to include the definition, I'll delete mine.

Too early to call, I ran it through the weekend and compared with Qwen3.8-Flash-Next, DSv41F-1017 at FP8 and Q3.8FN at Q8. Both were great, but surprisingly, Q3.8FN had the better result. I was modelling and generating simulation code for a complex system. I used both because I had a bunch of images that needed to be used for input to steer the output. I think these models have strengths and it's best to experiment and find what works of you. I didn't find MiMo2.6-Flash in the same tier, but MiMo2.6-Pro is a beast.

Reply on news.ycombinator.com