Distributed GPT training doesn't really work (sadly). Maybe someone will get a good solution to that, though.