On another note, digging in closer, the proposed training scheme (Section 4.3) sounds like it would be incredibly inefficient for any large neural models - you're computing your own updates via gradient descent, but also getting compressed updates from peers on the network, and then broadcasting your own updates out. For reference, at work I might be training a model with, say, 30M parameters, and I get about 4.5 training iterations per second on a GTX 1080. What would that drop to with peer broadcasting added in?
They mention only sending weight updates that under/over-flow a certain value, is that enough to avoid, e.g., a 5x drop in training speed? Given that training time is already a big bottleneck for ML work, this seems critical. Section 7.2 seems to address this, referencing Supplementary Materials (Appendix F). The numbers there don't look great - with the optimizations they mention, for GPU timing in Figure F.3 it looks like local training ops account for about 21ms, and the protocol overhead is something like 100ms. I also don't see an apples-to-apples comparison of "here's how long this example takes to train locally vs. with this protocol", which isn't a good sign.