Which is exactly why I compared sonnet 4.6 at the end there. They have similar intelligence scores.
Yes I know what the benchmarks say. It does not reflect my experience.
You’re also comparing benchmarks for the full model, but then telling people to run IQ3_S. I found the IQ3_S model to be a substantial step down from 4-6 bit quants.
> You’re also comparing benchmarks for the full model
I wish there were more concrete ways to judge the difference. I've spent a bunch of time using 3 and 4 bit versions of qwen 27b, and my conclusion was that the lower qaunt made more mistakes - but that both systems were able to recover from those mistakes in a coding context. I'm not sure if that holds across the board, but that's the best answer I have so far.
It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
With a $20/month subscription you can basically run Haiku nonstop and never have to worry about limits.
Your comparisons continue to make little sense.
> It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
This myth has been debunked over and over. There are numerous sites that run benchmarks against the providers repeatedly and the results are always within the same margin of error.
If all you do is generate tokens all day, sure, host your own infra. For people that aren’t maxing out hardware, using someone else’s is a pretty economical option. $20 with deepseek flash gets you a LOT of agentic use, and as the frontier moves forward with new models, so do I.
But let's compare apples to almost-apples! Qwen3.8 flash next is $0.47/M on openrouter, so $2.44 per day, so 246 days to break even. I say almost-apples because on this $600 system we're running an IQ3_S quantization. The hosted model is running... nobody knows what quant. It probably varies bases on the time of day.
My deepseek bill for the last 30 days was $2.91 for 354 million tokens for agentic coding. My heaviest usage month was $14.97 for a billion tokens at deepseek. If you are doing turn based agentic coding, 90ish percent of tokens or more are cache hits.
I have another use case for deepseek that does not hit caching at all, and I pay $2 for 12 million tokens every month. I could get that down by choosing slower providers, but speed matters for this use case and saving $.50 doesn’t.
Both those use cases are bursty. I can’t sit and wait for a server to chew through my requests, especially with my non-cache use case. Anything less than 200t/s output and I start looking for a different host.
As far as quants, direct from deepseek is full quant. Good providers tell you the quant. Even the budget ones typically won’t go as low as iq3, though. FP4 seems to be the lower limit of providers that I see.
I would love nothing more than for local models to be up to snuff, but for my purposes, they get blown out of the water on all fronts by api access. Your machine has a payoff measured in multiple years (assuming you can build it for that price: $300 worth of ram is not much unless you go for ddr3, and a $500 gpu is pretty minimal) and would be a really poor fit for what I’m doing. It seems to be a great fit for you, and that’s pretty awesome.