Towards universal neural nets: Gibbs machines and ACE
arxiv.org
arxiv.org
- Doesn't get to the point until about halfway through
- Repeatedly mentions Einstein
- Appeals to quantum mechanics out of nowhere
The style is particularly hard to parse (or I'm particularly dense, but I am generally comfortable reading papers on variational inference, neural networks, etc). At the same time, a lot of it rings true-ish... Does anyone get where they're going with this?
oh how far we have come
I could do with a few more recent citations on generative modeling. It seems the author isn't 100% aware of some of the most recent generative modeling work.
That said, the ideas presented are interesting and seem complimentary to lots of existing approaches - I will be looking into this paper further.
If I understand the paper correctly, the objective function is a combination of a classifier and a generative model. It sounds a lot like pre-training a neural network as a generative model before fine tuning it as a classifier, except this time the two steps are jammed together. I'm not sure what the benefit would be...
It's quite possible that I'm a fool, unable to see the beauty of the argument and the depth of the parallels being drawn, but to me it sounds like http://www.smbc-comics.com/?id=1957
Section 2.3 reads like a redefinition of the exponential family and its link to the maximum entropy principle. I think the idea is to optimize the sufficient statistics, but it's not clear.
Section 1.5... well you talk about two dimensional translational symmetries, and then make a link with position and momentum, but where is that coming from? This seems merely like an artifact of looking at two dimensional translational symmetries, and not at the general case. Again, what does the quantum analogy bring to the table? I think - though it's not clear - that what you're suggesting is drawing the latent noise following a distribution which is reflects the expected symmetry in the manifold.
So what is the takeout? Are you attempting to parametrize max-entropy distribution using the symmetries as constraints?