Foundations Built for a General Theory of Neural Networks

Deep Learning is currently an empirical science guided by intuition of practitioners. A main principle in experimental sciences is that a theory without predictive power is not considered a full-fledged theory. As such, unless they are interesting predictions coming from their theory (rather than only barely justifying existing empirically observed phenomena), this is just speculative theory that I would not use the phrase "Foundations Built" for.

As an example of this general litmus test for a theory see e.g. Eddington's confirmation of GR: https://en.wikipedia.org/wiki/Tests_of_general_relativity#De... . If there are hitherto unknown phenomena in DL predicted by this theory then I'd stand corrected and concede that there may be something to these theories.

I've beaten this drum before but another area to look at is network connectivity. When visualizing a neural network we typically see a fully connected network from one layer to the next, but if you set the value of many weights to zero you'll find that they have absolutely no effect on the function of the network. So basically we're looking at a lot of spurious interactions and this just clouds our thinking. In fact the network topology is what's driving network function here but we're not doing a good job exposing that intuitively. Computational research on Gene Regulatory Networks--which are also modeled with the same NN math--have shown how network topology is really the key driver of function.

For anyone who is interested, I published a paper during my PhD that has about 195 citations. It shows that under an evolutionary process that Artificial Gene Regulatory Networks select out spurious network interactions, leaving you with what's functionally necessary (Also see the supplementary). I came from AI before starting my PhD and believe that this could be important for building a foundation for NN as well.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/

Here are the two papers linked towards the end of the slightly rambling article:

https://arxiv.org/abs/1705.05502 https://arxiv.org/abs/1810.00393

I love the fact that I'm seeing how stuff works. Before theory, people built huge buildings, cars, airplanes, magical electrical circuits etc.

Neural networks are the same thing. Bunch of heuristics that will later be proven in theory. Yet still work incredibly well. Like cathedrals did without any mathematics.

Says a lot about how tinkering with heuristics is powerful.

The current state of neural networks is somewhat analogous to how cathedrals were built in the medieval era.

Master architects had a good understanding of then-ancient geometry and applied that knowledge plus knowledge of arches from Roman times to create the great cathedrals found throughout Europe.

The architects didn't have a deep understanding of material science (stone) and other forces that are taken into account in the modern era, but they had enough knowledge from experience and experimentation combined with relatively simple geometry to create strong structures that have stood the test of centuries.

The analogy between that and modern practical AI/neural network implementations is that experts in the field have good basic knowledge of how to construct neural networks but there isn't yet an advanced theoretical knowledge of the science that allows for precise predictions and modeling. i.e. Experts are still at the level of experimentation and noting the results and getting useful results without a super-deep understanding of how neural networks produce useful results.

Hypotheses have been/are being proposed and tested right now. Expect that as a deeper level of experimental results are achieved we'll soon have a sound theoretical model that will lead to profound advances in the field.

I think a important advance is being able to build neural networks in a modular fashion by constructing a toolkit of components and techniques one can reasonably expect, when combined, to produce certain result. As the field advances, this toolbox will become richer and better defined and delineated, forming basic building blocks. Just like functions, loops and virtual methods came out of the chaos of assembly.

Proof of the universal approximation theorem [0] for those interested

[0] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441...