heyitsguay is correct, but in addition:
- The researchers didn't even explore quantization. In theory 4 bit quant would allow for training on even more modest hardware.
- Memory use aside, forward pass only is potentially a big training speed increase.
- This method is very amenable to decentralized training.
It scares me because it feels like a gateway to networked, self training LLMs on commodity hardware. I thought this was a long way away... now it doesn't feel so far away.