The problem is always RAM. On Apple Silicon RAM is shared between CPU and GPU, which means a 64GB of RAM machine can run really big models with GPU acceleration.
On Windows/Linux machines you usually need to consider the VRAM available to your GPU. I haven't got a great feel for what kind of VRAM numbers are common for different consumer setups there, so I don't tend to write much about those.
Of course llama.cpp will happily run models on CPU as well, but again I don't have a great feel for how usable that is.
I think the main impediment is Intel and SYCL not working on all their cards... AMD is good to go, I think.
Btw, this is my favorite laptop of all time. Highly recommend it.
It’s painful, but the results are night and day compared to the models that will fit entirely in vram. Of course perhaps I’m doing something wrong so if others have advice it would be great to hear!
I have a 9800x3d/48gb ddr5/3090 and this model still being ~40GB when quantized is a challenge. Sure would be nice to have a 30B version that can quantize down to ~16GB. Or I guess I could upgrade my PSU and get another 3090 and use NVlink...
It’s the throughput. For whatever reason they’re a good 5-10x lower.
Even if it’s a large amount and gpu accessible if the throughput sucks that badly then 70B models aren’t usable
What puzzles me is that the even the newer AMD APUs have this issue. The upcoming strix halo is rumoured to be around 140 ish which puts it at very bottom end of Apple devices
Windows machines with dual channel DDR5 are only around 80 GB/s bandwidth.
Nvidia cards are around 1 TB/s, but models with 40+GB of VRAM are more expensive than macs with equivalent RAM capacity.
I would expect that an Nvidia card on Windows would be around twice as fast as apple silicon if you have a model with 40+GB, and from some quick research this seems to be true.
edit: "Any Macbook 64GB can run it at about 6-7 Tok/s using Q4 quantization. 2x3090 can do it much faster at about 18 tok/s" https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment...
The actual ram is lpddr5 - miles slower than what you’d find in even very old dedicated GPUs
It’s as you say though - for some tasks like LLMs that doesn’t matter if you’ve got enough bandwidth to said slow ram…ends up being almost the same thing as having actual fast ram.
It’s a neat trick & little bit frustrated that nobody is copying it. Cause using dirt cheap lpddr5 (about a dollar per gig) but paying Apple’s comical ram upgrade prices feels like a cruel joke.
Rumors are there may be a X3D Threadripper in the works, would be nice to not have to choose between RAM bandwidth and IPC again.
So, neat, I guess, but not convinced I would pay four macbook airs for that.