For somebody who hasn't kept up with all the different open source models and what token rates mean, how would using this compare to something like codex or claude? I assume it would be much slower and less capable..?
As sibling said, you won’t hit the same quality using a 2 bit quant like this project, which is unfortunate but that’s the reality today.
The ~issue is though that they're using a pretty low quant, 2 bit. 4 bit are _pretty_ close to full ability, maybe degraded a bit. 2 bit is _noticably_ worse. It's about as small of a quant as you can get while still pretending it works, for this model.
This won't replace claude for you, unless you baby it a lot and are okay with it being kind or derpy.
However you can run comparable or better models than gpt4. Try to stay under 30b