The new MMAP system would not require you to load the whole model from disk up front, but rather would allow you to load and immediately forget each layer of parameters as you do a forward pass through the GPT architecture. And if you had just tried to use normal swap paging to do this without MMAP, you'd essentially be doing 2x the loading work, since the page for the first layer likely got swapped out as you loaded the rest of the model into "memory."
Of course, as soon as you want to generate a new token, you'd need to reload all the pages of the model parameters. Every token generated would take as long as it would if you were loading the model from scratch.
But for a classification problem where you only need to generate one token, and you need or want to do this in a serverless environment where you'd need to load from disk anyways? I think you'd be able to do that workflow with significantly less RAM in the same amount of clock time, now.