If the model is all in RAM and being moved one layer at a time to a GPU, would that be faster than getting it to the CPU? What about if the model is on an SSD. The point being, compute is not the bottleneck so I wonder if this provides speedup compared to an optimized model run on a CPU?
1. The GPU was still faster, though the difference was smaller than I really expected.
2. Almost all the time was spent on compute, not loading weights to/from the GPU. For models I could fit on my GPU the difference in performance in loading one layer at a time vs loading them all was completely negligible, and the difference in memory usage (constraining what else I could do with the computer simultaneously) significant.
3. Loading from SSD was a huge bottleneck when I tried that
Hardware: RTX 2070 super, 5800x
The way to get a speed up is to split the load across both the CPU and GPU since LLMs can be split into layers. This also increases the size the model can be.
Even for non-batched, you can still optimize this a bunch for fairly big models without much more work. Basically, overlap io & compute: while one layer is computing, already be loading in the next layer(s) in parallel. PCI speeds are in the 1GB/s-32GB/s each way for consumer cards nowadays.
Without a lot of work, they might be able to fully hide the IO latency.. very model + hw dependent.