Hacker News Reader

Comment by Roark66 | Hacker News Reader

I hope popularity of large models like this one drives more work on CPU inference of quantized models. It is extremely disappointing one can't run 4 or even 8 bit quantized models on a cpu. Inference I did with fp32 on a last gen AVX2 CPU show me it is definitely usable if you're willing to wait a bit longer for each token (I got about 1token per 2s on a ryzen 3700x, 32GB ram, with falcon-7B-instruct and this is with about 1gb of ram in the swap).

I don't quite understand why people aren't working on cpu quantization. Allegedly openvino supports _some_ cpu quantization, but certainly not 4 bit. Bitsandbytes is gpu only.

Why? Is there any technical reasons? I recently checked and for a price of a 24gb rtx3090 I can get a really nice cpu (ryzen 9 5950x) and max it with 128gb of ram. I'd love to be able to use it for int8 or 4 bit inference...

View on HN

>It is extremely disappointing one can't run 4 or even 8 bit quantized models on a cpu

What do you mean? Llama.cpp can do 8 and 4 bit quantisation on CPU, and even supports Falcon 40B.

Yes on arm. Sorry I meant the cpu (x86_64) not a cpu. :-D

Its quite fast on x86.

llama.cpp looks really good on Mac ARM CPUs because:

- they have tons of memory bandwidth

- they have a really good proprietary acceleration library (accelerate)

But I don't think it would be so fast on, say, an Ampere Altra compared to a similarly priced EPYC cpu.

Ampere has native acceleration (Ampere Optimised Frameworks) for AI workloads

I dont think llama.cpp integrates this though.

Accelerate is all but depreciated because of the Metal backend anyway.

These things are quite memory bandwidth bound, so if you are getting any kind of machine to run them go for the absolute fastest RAM you can.

I told you a while ago that llama.cpp supports 4/8-bit (and other) quantization just fine (and fast) on x86_64.

Yes, you did. Back then I went to their github page and I found no mention of x86. Now I see it is there. Great. I'll give it a try.

Also I see something much better. The same guy behind llama.cpp authors a universal ml library that does cpu quantization. I wasn't aware of that and it is far more impressive than running a couple of chosen models.

https://github.com/ggerganov/ggml

TinyGrad is also targeting CPU inference, and IIRC it works ok in Apache TVM.

One note is that prompt ingestion is extremely slow on CPU compared to GPU. So short prompts are fine (and tokens can be streamed once the prompt is ingested), but long prompts feel extremely sluggish.

Another is that CPUs with more than 128-bit DDR5 memory busses are very expensive, and CPU token generation is basically RAM bandwidth bound.

because on a per-inference level, it's _still_ cheaper to use GPU than it is to use CPU, even if you ignore latency and only consider offline batch applications.

Or, inference that can fluidly use available CPUs, GPUs, and neural units on M1/M2/M3.

Not if a 96 GB GPU costs $30K?

The OP notes that they can infer a 7B model at 2s/tok using their CPU setup. Equivalent inference on a 4090 ($1600) is up to 400 tok/s, or 0.0025 s/tok. This is an 800x difference.

(to be fair, you can do better than the OP's performance, but we _are_ talking several magnitudes of performance, not to mention energy efficiency, which matters at scale)

> Equivalent inference on a 4090 ($1600) is up to 400 tok/s

What? Who's doing 400 t/s @ 7b on a 4090?

I am, with many optimizations eg dynamic batching, tensorrt, but most important is batch size > 1 -- cheating a little bit in the comparison ;).

batch size 1 is more like 200 tok/s, still a very large improvement. the reason why this is still an interesting comparison is that it's comparing consumer hardware vs scaled datacenter setups. datacenter setups are much more efficient in large part because they are designed to process many inputs at once, fully utilizing the hardware. there are advantages to scale in inference, for the gpu-rich.

I’ve been using llama.cpp to try out some large-ish models on my cpu for a while now. I’m on an amd 3600, with 32gb ram.

I can run 30B 4bit and 5bit models. It’s definitely “usable” but smaller models definitely run faster.

I don’t have a good graphics card, but it does work on cpu. With more ram I think the 65B and larger models are runnable as well

I would be willing to bet Apple will be one to invest in CPU-based work in this space

Reply on news.ycombinator.com