Hacker News Reader

Comment by svnt | Hacker News Reader

“This new technique” is they load each layer into memory, process the result of that layer, and then clear it and load the next layer. For the cases where this even works, as one commenter said:

“I try it on a GTX 1060 6Gb on windows, but I don’t think it’s supposed to be that slow? It took me 13 hours to generate one sentence, RAM was around 9.5Gb and only 6% of GPU in use. It works but I don’t know what I’m doing wrong”

View on HN

> “This new technique”

Which also has a name that they even use, model sharding. Which is well known to be very slow because you increase the number of IO operations to the GPU. I don't even understand the post because they mention sharding is slow and then say they do sharding but as if they don't. Even the comment in their code calls it sharding. Unless I'm missing something, it's pretty fucking deceptive to call this a new technique. Can we stop with the AI hype? The field is exciting enough that we don't need to be selling snakeoil and selling everything as if it is 10x better than it is.

For anyone interested in model sharding you should look into pytorch's gradient checkpointing documentation and/or fairseq.

To echo and in no way contradict the parent’s point, there have been versions of this going on forever.

Without getting into a bunch of stuff about FDSP or Accelerate or even the NCCL/All-Reduce type ops (which I obviously hope the interested check out) I’ll signal boost my favorite tech talk ever by maybe the coolest CS prof living:

https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1

I’m biased because I’m also Ben, but I’m not “Ben Rekt” who is a musician and charismatic and way too cool to care about SV.

Btw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts

Hogwild is a really cool routine. There's a lot of work going into FDSP, parallelism, and all kinds of things at the low level for optimizing communication, maximizing parallelization, writing cuda kernels, and so on. I think there's a lot of unsung heros of ML working in these areas that people never even see (including researchers). So many people quote Sutton's Bitter Lesson but these are the people that made that landscape even possible. Without these people working at the low level we wouldn't be able to process large models or large datasets.

The fame issue is always a bit weird to me. Like I think more people know of Vaswani (AIAYN) or Dosovitskiy (ViT paper) than those that know Kingma, who wrote the Adam optimizer. Both these works use Adam as well as Rombach's LDM (Stable Diffusion). I'm only mentioning Kingma because I'm more aware of that work but am certain there's tons of work that could be better highlighted (actually illustrating this point).

I was playing with this with the llama model was released - and provided I was loading from memory and not disk it was actually only negligbly slower than the model was without sharding (for e.g. the 8gb models that I could load entirely on my GPU). Each layer took a lot more time to evaluate than it took to copy the weights onto the GPU (RTX 2070 super).

Yes, if you can fit it in RAM then the loss isn't too bad if you have the memory pinned and are taking advantage of some of Nvidia's optimizations (which should be defaulted to on). Inference usually isn't too bad but training is heavier. Things have also massively improved over the last few years. The efficiency also really increases with reduced floating point types and quantized models.

Yeah, I'm sure the performance of this must be pretty bad. As I understand it, non-batched LLM inference is generally limited by the bandwidth required to read the entire weights from GPU memory for each token produced. This trick instead moves the bandwidth bottleneck to loading those weights into the GPU from wherever has enough space to store them, probably some kind of SSD, and that has much lower bandwidth than GPU RAM.

For CPU inference, performance is generally memory bandwidth limited. A modern cpu can do the matmuls (all of which are matrix-vector) faster than the data can be loaded in from memory.

If the model is all in RAM and being moved one layer at a time to a GPU, would that be faster than getting it to the CPU? What about if the model is on an SSD. The point being, compute is not the bottleneck so I wonder if this provides speedup compared to an optimized model run on a CPU?

Tried this awhile ago on the llama models, with my hardware

1. The GPU was still faster, though the difference was smaller than I really expected.

2. Almost all the time was spent on compute, not loading weights to/from the GPU. For models I could fit on my GPU the difference in performance in loading one layer at a time vs loading them all was completely negligible, and the difference in memory usage (constraining what else I could do with the computer simultaneously) significant.

3. Loading from SSD was a huge bottleneck when I tried that

Hardware: RTX 2070 super, 5800x

Just want to add, this approach is the exact opposite of conventional wisdom around cache hierarchy. Here we are loading data in from the slowest source, using it once, and throwing it away. I'm not sure what can be changed with a sequential model like and llm, just that this is almost the textbook slowest way to run the model.

It would potentially be slower because pcie bandwidth is likely to be lower than memory bandwidth.

The way to get a speed up is to split the load across both the CPU and GPU since LLMs can be split into layers. This also increases the size the model can be.

It sounds like they do it naively, which is fair for a POC

Even for non-batched, you can still optimize this a bunch for fairly big models without much more work. Basically, overlap io & compute: while one layer is computing, already be loading in the next layer(s) in parallel. PCI speeds are in the 1GB/s-32GB/s each way for consumer cards nowadays.

Without a lot of work, they might be able to fully hide the IO latency.. very model + hw dependent.

I appreciate your and the other commentators scepticism, and agree for the premise of generating text. However to play devils advocate - Can't inference of single layers be quite useful sometimes for embeddings? and the larger models could embed capabilities only achievable at large parameter size? (i.e. you don't need to finetune the last layer like the BERT days or necessarily use a certain layer for relevant embeddings, you could use the first layer outputs and finetune a custom matrix projection to make some useful embeddings that might include properties of the input that the larger models show capabilities for that smaller models don't?)

(ofc this assumes that those capabilities considered "emergent" at larger sizes do not require later layers which they probably do). There is a weird situation where we have the sentence transformer, the bge, e5 and etc kind of embeddings, and then a big jump up to generative model embeddings the model providers provide, but not much widespread adoption of e.g GPT-J or neo-20B embeddings (even though at time of release, they had notable usecases over sentence transformers)

There must be some real use for it for sure. Maybe as a result of this effort someone else has their attention called to it and picks up the idea loosely and runs with it in an unexpected way.

One of the benefits of lack of contextual or historical understanding is the lack of barriers to action. One of the downsides is you’re likely to do a lot of things of questionable value.

You can save some space, but it still takes more than one layer to do an embedding.

> 13 hours to generate one sentence

This is hysterical, they're literally swapping from VRAM to RAM and certainly to disk as well since a 70B GGUF is at least 30-40 GB in total.

I suppose it's slightly better than not being able to to it at all, but jesus christ this is just hilariously bad.

Reply on news.ycombinator.com