So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU.
Sure the GPU i not fast, but if memory is important.....
So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU.
Sure the GPU i not fast, but if memory is important.....
And for the moment, its slower than pure CPU. Optimizing for IGPs is not trivial.
MLC's Vulkan backend is actually quite good on my AMD APU, but unfortunately it won't split the model to a dGPU.
M1 Pro is 200 and M1 Max is 400. Which is slow for GPU memory, but incredible for main memory -- although I'm not sure how much of that a single core can actually pull.
The author (correctly) made a distinction about the two like you said, but at the end when talking about Raspberry Pi 4 they use a number (~4GB/s of memory bandwidth) from an article [1] which I can only assume is NOT about graphics memory or its bandwidth (do Raspberry PIs even have it?).
And how exactly is the bandwidth counted if I use integrated GPU (like i5 13600K)? Or pure CPU?
An integrated GPU shares memory with the CPU, so at best it gets the same amount of bandwidth assuming the CPU is not using any (which is rather unlikely).
A dedicated GPU has its own private high-bandwidth memory (an RTX 4090 has a memory bandwidth of over 1000 GB/s), but to get anything in there it needs to be loaded over the PCIe bus (which has a measly 32 GB/s bandwidth for PCIe 4.0 x16).
That said, CPU memory does have one big advantage: it tends to be much larger. You can pair up to 192 GB with a regular Ryzen 7000 CPU, while consumer GPUs don't go above 24 GB of memory (RTX 4090, RX 7900 XTX). (There are bigger GPUs out there, but those are generally intended for datacenters, and if you go that route, an Epyc or Xeon CPU can also support much more memory than a plain desktop Ryzen, although you can also slot multiple GPUs into a single server.)
I believe that for LLM performance, memory bandwidth is key, because all the neural network layers need to be streamed from memory, and very little work is done with it each time, although I guess batching operations could help if you're working at scale, since each weight would be applied multiple times then.