Comment by versteegen | Hacker News Reader

Comment by versteegen | Hacker News Reader

Parent Full thread

It's surreal to see this on the front page since I worked on minimax entropy learning for texture modelling (following these authors), then abandoned that line of research because I wanted to work on concept induction and reasoning instead. I'm amazed these authors are still drawing a direct line through 27 years of their papers.

To explain what this is about (not having read the whole paper): it's using 'classical', small sample size statistics to infer relations between very small sets of data such as individual questions of an IQ test. Obviously that's going to give the best possible answer, because that's what classical statistics does. (Well, if you could solve exactly, but with this model you actually can't.) Then it's just a question of having a good prior over possible concepts. In this paper the concepts are hand-defined parametrized features (aka 'filters' in the paper, from the texture modelling lineage), e.g. "families of arithmetic expressions of length from two to six" for the number problems. There will be a lot of details involved in that.

Alternatively, something like a deep neural network trained on vast numbers of generated examples could come to learn what a suitable prior should be by learning the patterns used to generate them. Yet another option is to use a universal prior as in Solomonoff induction, based on representation/coding length.

My conclusion: the theory is good and I'm happy to see this but don't think that you can directly apply this to learning 'concepts' in anything other than IQ tests. The hard part of generalising this is firstly finding the prior over the concepts, and secondly the difficult optimisation step: using MCMC as part of the optimisation is both extremely slow and stochastic in a bad way (unlike SGD) and the reason I abandoned this approach.

deep neural network trained on vast numbers of generated examples could come to learn what a suitable prior should be by learning the patterns used to generate them.

This sounds promising. Do you see any potential issues with this approach?

DNNs are very good at learning features so are a natural fit if you have lots of training data. The main drawback is then you've lost the ability to work with small data. The other drawback is the computational cost of the maxent model on top of it. So you get both the best and the worst of both halves.

So:

input --DNN--> feature vector --minimax_entropy_optimisation--> maxent model --sampling/inference--> output

But minimax entropy learning is just learning a model that matches statistics of a set of features (filters). If you just want to draw a single sample from that model or the maximum likelihood point (e.g. you just want the answer to an IQ question) rather than doing complicated inference (in the DNN era, people have forgotten that "inference" is far more general than classification) you don't actually need the model at all, you can skip learning it, and go straight to computing a result with the desired statistics. This is around 100-1000x faster.

input --DNN--> feature vector --statistics_matching--> output

This is what I mostly did in my PhD work, and there's a famous paper doing this:

A Neural Algorithm of Artistic Style https://arxiv.org/abs/1505.07376

This has a sister paper which is even purer statistics-of-features-matching:

Texture Synthesis Using Convolutional Neural Networks https://arxiv.org/abs/1505.07376

However soon people worked out you can just train a DNN to do the statistics matching directly. Initially, with a fixed target style, and later taking the style as a second input. If the results are acceptable, this is vastly faster at inference time.

input --DNN--> output

Using them in this way would likely mean just replacing the final layer. The original papers on 'style transfer' did this, a DNN plus minimax entropy learning. has been done in texture and image models.

(Whoops, ignore that last line; just fragments of unfinished sentences)

This is how NN theory describes the process of training, i.e., that's the mechanism by which all neural networks come to represent the problem they are being applied to solve. GP is using a fancy way to say "train NNs on massive datasets" which is par for the course.

Reply on news.ycombinator.com