Yes on arm. Sorry I meant the cpu (x86_64) not a cpu. :-D
llama.cpp looks really good on Mac ARM CPUs because:
- they have tons of memory bandwidth
- they have a really good proprietary acceleration library (accelerate)
But I don't think it would be so fast on, say, an Ampere Altra compared to a similarly priced EPYC cpu.
Accelerate is all but depreciated because of the Metal backend anyway.
Also I see something much better. The same guy behind llama.cpp authors a universal ml library that does cpu quantization. I wasn't aware of that and it is far more impressive than running a couple of chosen models.