>It is extremely disappointing one can't run 4 or even 8 bit quantized models on a cpu
What do you mean? Llama.cpp can do 8 and 4 bit quantisation on CPU, and even supports Falcon 40B.
What do you mean? Llama.cpp can do 8 and 4 bit quantisation on CPU, and even supports Falcon 40B.
llama.cpp looks really good on Mac ARM CPUs because:
- they have tons of memory bandwidth
- they have a really good proprietary acceleration library (accelerate)
But I don't think it would be so fast on, say, an Ampere Altra compared to a similarly priced EPYC cpu.
Accelerate is all but depreciated because of the Metal backend anyway.
Also I see something much better. The same guy behind llama.cpp authors a universal ml library that does cpu quantization. I wasn't aware of that and it is far more impressive than running a couple of chosen models.