For those of us with Twitter.com blocked in local dns, what size model is it getting 40tok/s on?
here's the linked github fwiw
> Watching llama.cpp do 40 tok/s inference of the 7B model on my M2 Max, with 0% CPU usage, and using all 38 GPU cores.
> Getting 24 tok/s with the 13B model
> And 5 tok/s with 65B
Which is still incredible. I recently tested llama.cpp on a dual xeon server with 24 cores and 70gb RAM and on the 65B model it took like 15 seconds for one token.
GPU seems the way to go here