Hacker News Reader

Hacker News

Comment by taeric | Hacker News Reader

Parent Full thread

taeric

I'm curious on this claim. Feels like any model can over fit.

View on HN

A model can only overfit if it has enough parameters. But indeed, any model with enough parameters can overfit.

The thing about deep learning is that the number of parameters is astronomical.

Hence, the interesting question is why deep learning models don't overfit all the time.

This is from 2019 so I suspect there’s a more recent paper now, but this is interesting: https://openai.com/blog/deep-double-descent/

Agreed, but it doesn't really explain it in any real sense.

“However, in the over-parameterized regime, there are many models that fit the train set and there exist such good models.”

One explanation is that a very large model effectively fits several models and penalises over-fit models itself.

Work is ongoing, this is a whole field of research now, a good keyword to search for is "implicit regularization".

I really like how AI research is all about empirical analysis of mathematics.

The answer is really very simple: because we use training and test sets, and generally use iterative training methods. As soon as test set error plateaus or starts to degrade, you stop training. This is called early stopping and can be considered a form of regularization. In theory it would seem a neural network (with early stopping "correctly" implemented) can never overfit, although I suppose in practice it can vary a little (because of test variance, too large step sizes, etc.). It's possible to train huge overparametrized networks this way.

One big problem I can see with this approach (maybe compared to over forms of regularization) is that it might be very space/memory inefficient. But if you've got a big budget I think it's a surefire way to make sure your model performs as well as possible (without having to tune hyperparameters too much or deal with local minima that probably happens at near-capacity networks).

Random Forests can overfit, but with sufficient data (and no data leakage) it is difficult to do so.

The final output of a random forest regression for example is just the average of all the final prediction nodes of the individual trees, which thanks to bootstrapping are decorrelated. So - adding more trees does not tend to alter predictions very much.

With algorithms that, for each additional input entry, either add a predictable amount of complexity to improve fitness (e.g. another tree split) or decide that the model is already fit enough, one needs to do something deliberate or very stupid to overfit data compare to neural networks where overfitting is hard to detect, let alone prevent quantitatively, and that are sometimes designed to overfit in order to interpolate "intelligently" between samples.

Reply on news.ycombinator.com