Most people don't realize that what is popularly known as "deep learning" has actually been around for over 50 years now. The big gains are really the result of having additional computing power to throw at the problems.
Of course GPU-based training has helped a lot, but my impression is that these theoretical breakthroughs were also important.
The improvements in techniques make new discovery faster and news discovery create better techniques.
Interesting times...