DeepSeek V4 with 1 million token context window is pretty powerful, although still not there. There's hope that Opus 4.5 level performance locally is not that far away.
Running DeepSeek V4 without extreme quantization locally requires a lot of hardware.
The IQ2 quants that fit into 128GB machines are very degraded.
That is true, it is a 1.6T parameters model so it requires a great deal of memory. I also heard there's a 2bit quantization that works well on Apple metal.
From what I read, ds v4 is very close with opus 4.6 performance.
The full model is, not the quantized versions.
yeah that goes without saying. how can openweight, quantized version beat SOTA :)
Well it depends on the task. For agentic coding, more is more, but for tasks that normal consumers use them for there really is a ceiling. OCR, text to speech, that type of thing doesn't really improve when going to a SOTA model, so you'd just be wasting your money. I think local LLMs have more value than software engineers give them credit for.
totally agree with that. local llm doesn't need to match SOTA performance in order to be useful.