We double and triple and quadruple checked our results, to make sure that we are in fact getting results like this while only doing our thing every step, and it really keeps holding up.
Don't trust our word for it, though, you'll see when the paper comes out :)
I could be less harsh if this was some grant requirement to release a report before a certain date, but I don't see any grant funding declaration.
I've not kept up-to-date with training/optimizer research for quite some time but during the deep learning craze there were papers like the ones about DistBelief/Downpour SDG[0] that showed how to scale up training by only doing occasional synchronization. Did that not transfer to transformer/LLM training?
[0]: https://proceedings.neurips.cc/paper_files/paper/2012/hash/6...
The linked PDF is very light on detail, but what results they do claim are about a 1.2bn parameter model. This is tiny; you don't need network-bound distributed training (ie, anything beyond a single datacenter class machine, or less if you're patient) to train a model that size. The comms requirements also scale with the model size, so I strongly suspect people hoping for embarrassingly-parallel-style scaling properties are going to be disappointed.
(They also appear to have, in part, reinvented parameter servers.)
without more details it's unclear if the proposed technique keeps its speedups in that case