Depends on the architecture, but generally you just move through the layers linearly. Simple iteration.

The number of layers, and the amount of time spent in each of them, makes me think any benefit from pre-loading the layer ahead is negligible.

You really need the entire model on device to consider it performant.

This isn't how neural networks work.

For vanilla models, you always use all the weights. That isn't true for mixture-of-experts, though, and in that setting, your approach has merit.

That looked like an analogy. Back in the days of a mechanical arm moving magnetic fields around in our PCs, you could have the illusion of infinite RAM as long as you're ok with microsecond operations now taking two million times longer. This is akin.

I think the point is that it has the same sort of latency tradeoff that disk swap did: it's awful, but sometimes better than nothing.