Learning@Home using Decentralized Mixture-of-Expert models:

https://learning-at-home.github.io/

https://training-transformers-together.github.io/

https://arxiv.org/abs/2002.04013