Hacker News Reader

Comment by sp332 | Hacker News Reader

Man, it's good to see some actual information theory applied to deep learning for a change.

I'm sorry but this comment is just at odds with the current literature in deep learning. A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'....

https://arxiv.org/search/?query=deep+learning+information+th...

I think the GP poster meant to contrast this paper not with other papers, but with the junk explanations for "how deep learning works" that science journalism comes up with.

trust me some of the junk explanations for how deep learning works originate from researchers. we really just don't have a convincing picture about why these networks work or don't work in a theoretical way rather than empirical/heuristic handwaves

Bollocks see my comment. We know worst case bounds for Kolmogorov random transformations of a finite set, the most complex learning task there is.

huh? not even wrong......

Good. I hope to see more of them posted to HN.

if you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening... but some of his key claims are probably contradicted by empirical results.

the original paper: https://arxiv.org/abs/1503.02406

a refutation:(with very harsh/unprofessional words by tishby in counter-refutation)

https://openreview.net/forum?id=ry_WPG-A-&noteId=ry_WPG-A-

I found his talk on youtube, remarkable:

https://www.youtube.com/watch?v=XL07WEc2TRI

He's a good speaker/teacher but I think there are serious problems with his work- mainly that a lot of his results collapse when tested on more general architectures.

There's papers posted to arXiv every day using information theory to analyze deep learning...

Reply on news.ycombinator.com