Hacker News Reader

That's to be expected. 4 bits is too small. But the 8 bit 65b should outperform significantly.

Have you tried it?

Is 4 bits too small? Or is the quantization method just not sophisticated enough.

I’ll give it a try this evening.

post example prompts and results please?

From what I can tell, the quantization being used there is extremely naive.

If they were using techniques like GPTQ, presumably it would work much better.