this seems like the trivial, most obvious way to parallelize training across GPUs. Not, imo, clever.
The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
Plenty of theoretically trivial solutions to problems are absolute pains to implement. I mean there are entire companies that at their core solve relatively "trivial" problems but employ huge numbers of engineers. Just because the core concept is simple to explain doesn't mean it's easy.
https://news.ycombinator.com/item?id=14510146
So the fact that someone with little experience can come up with this 'clever' technique, means either I'm really clever or it's not that clever. I'll go with the latter.