Most of the recent advances in ANN learning came from using more data and using different types of layers. A real theory of neural nets would explain why architecture A works better than architecture B for a particular problem. Or why a certain architecture plateaus after working with X samples, while other can learn from many more.
The way I understand it, the core problem of understanding deep learning is that it does not scale down. You can make a simple framework of 3-4 neurons and observe how it works. Then you make it bigger and it works differently. No one properly explained why this happens. People who do applications seem to just state their intuitions and then throw a couple formulas on top to make it seem scientific. There is a great essay on that by Piekniewski[2]. No research I've seen lists all the parameters/architectures/datasets they tried that didn't work.
[0] - Maybe with the exception of the paper on neural networks as ODEs.
[1] - http://neurophysics.huji.ac.il/sites/default/files/PhysRevA....
[2] - https://blog.piekniewski.info/2018/10/13/deep-learning-the-w...