What innovations do you have in mind? LSTMs and convnets were both invented in the 90s. Stochastic gradient descent, momentum, etc. have been around for decades. I feel like it's quite the opposite of what you are saying -- in the 2000s there was a lot of innovation, but it mostly turned out to be unnecessary.

What has improved is the variety of architectures, but I don't think there has been anything fundamentally new after LSTMs. The only exception I can think of is generative adversarial networks, and even ideas behind adversarial training have been around in other domains for many years now.