tl;dr they found a clever way to spread the training across 256 GPUs by synchronising the stochastic gradient descent
The important bit here is that they've shown that large mini batch sizes still can maintain accuracy if you slow the learning rate.
https://news.ycombinator.com/item?id=14510146
So the fact that someone with little experience can come up with this 'clever' technique, means either I'm really clever or it's not that clever. I'll go with the latter.
Plenty of theoretically trivial solutions to problems are absolute pains to implement. I mean there are entire companies that at their core solve relatively "trivial" problems but employ huge numbers of engineers. Just because the core concept is simple to explain doesn't mean it's easy.