These performance numbers look absolutely incredible. The MoE outperforms o1 with 3B active parameters?
We're really getting close to the point where local models are good enough to handle practically every task that most people need to get done.
We're really getting close to the point where local models are good enough to handle practically every task that most people need to get done.
After trying to implement a simple assistant/helper with GPT-4.1 and getting some dumb behavior from it, I doubt even proprietary models are good enough for every task.
You should try a different model for your task.
Seems like 4 bit quantized models would use 1/2 the number of billions of parameters in bytes, because each parameter is half a byte, right?
Again, very rough numbers, there's calculators online.
Of course, there are lots of factors that can change the RAM usage: quantization, context size, KV cache. And this says nothing about whether the model will respond quickly enough to be pleasant to use.