New Theory Cracks Open the Black Box of Deep Learning
quantamagazine.org
quantamagazine.org
The information bottleneck idea is very similar, basically the same, as how I've always thought about DL models, and statistical models more generally. The hidden variables at each layer are basically digitized codes, and there's a compression at each layer, which is equivalent to learning/inference as in an algorithmic complexity/MDL sense.
What was surprising to me was the relationships with renormalization groups, which I wasn't familiar with at all.
The quote from Lake was also interesting. I forgot about that Bayesian Program Learning paper, which was interesting. My guess is BPL and DL are not really all that different at some level.
This would explain the success of attention mechanisms quite well. If you are good in selecting the right input you don't need to discard a lot later.
I particularly enjoy attempts to understand basic black box mechanisms in the space of learning or cognition.