A naive GPU implementation on a non-block sparse network will be just as slow as the full un-pruned network.
Another approach would be to lower precision. By going from 32bits to 16 or 8 you can get massive speedups in memory operations and computation.
Yes, that'll work. But it's orthogonal to this paper.
The trick is to combine these 2 approaches. Quantizing to bytes helps a lot, but forces you to use much larger pruning group size to see any speed up from pruning.
Nvidia's apex.amp does just this - attempts half-precision training where possible, and adjusts scaling factors during training. I achieved a very significant speedup using this on my RTX 2080 Super, though it doesn't support Windows that well.