> What makes one kind of neural net 'deep' and are all the other ones suddenly 'shallow'
Number of layers
It's that simple
Number of layers
It's that simple
Eventually, people realized that you could improve performance by adding more hidden layers. So while theoretically Cybenko was correct, practically stacking a bunch of hidden layers made more sense. These network architectures with stacks of hidden layers were then labelled as "deep" neural networks.
[1] https://en.wikipedia.org/wiki/Universal_approximation_theore...