Currently running 65B on my 96GB M2 Max.. it's pretty good.
Cool if it's able to run 100% CPU-based because that makes portability and deployment a lot easier. Makes this code a lot more accessible.
I previously tried to play with ANE based off prior reverse engineering work [0] but couldn't get it to work nicely. It's actually beyond me how Gerganov's version performs so well -- the output quality of the non-quantised version running on A100 (AWS) isn't noticably better than the one I'm getting.
[0] https://i.blackhat.com/asia-21/Friday-Handouts/as21-Wu-Apple...
Plan is in the future to utilize respective SIMD intrinsics for other architectures (AVX, WASM SIMD, etc) and also add other more accurate quantization approaches. It's actually not a lot of work and I have most of the stuff ready, so hopefully soon!
Edit: AVX2 support has just been added
I just look at the code and port it. I don't have the hardware to run it.
If you need extra hardware I'm sure the community could make that happen.
(20 tokens/second on a Mac is for the smallest model, ~5x smaller than 30B and 10x smaller than 65B)