Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading?
Please do correct me if I’m wrong.
> The first time you load a model after rebooting your computer, it's still going to go slow, because it has to load the weights from disk. However each time it's loaded afterwards, it should be fast (at least until memory pressure causes your file cache to be evicted).
* 2x memory. A 20G data set requires 40G (20 for page cache and 20 for LLaMA)
* Things would be _even slower_ if they weren't in page cache after first loading. mmap is fast because it does not require a copy and reduces the working set size