> the gains here are mostly due to not copying memory anymore, and better cooperation with the kernel's page manager. We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved.
If I understand correctly, this only provides a speed up when the model is already in the OS’s file system cache, and in any case you still have to load the entire model into memory.
But if the intent is to have multiple processes share the samedata then mmap likely is the better choice.
With a GDDR6X memory bandwidth of 936 GB/s and PCIe 4.0 x16 bandwidth of 64GB/s loading something like 20GB into the VRAM of an RTX 3090 shouldn't take longer than 1/2 a second or so, right? (assuming it is in the kernel cache)
https://github.com/ggerganov/llama.cpp/issues/91#issuecommen...