Estimating Coefficients in Linear Models: It Don't Make No Nevermind (1976) | Hacker News Reader

Estimating Coefficients in Linear Models: It Don't Make No Nevermind (1976)

www-stat.wharton.upenn.edu

36 pointsnkurz13 comments

Any commentary on why would anyone consider this paper of interest today? The peculiarities of the least squares fit are no secret, the assumptions of the "generalizability of fit theorem" presented are highly unrealistic (compares equal weights to weights chosen uniformly at random), and we are now living in the age of things like Support Vector Machines, Structural risk minimization, VC dimension, boosting and so forth, which all address the issue of overfitting very well. This reads like prehistory of the field, and not very significant at that.

I think there may be more substance to the concept of equal weighting than there appears to be at first glance. I came across this paper via a blog post from Andrew Gelman. In that post he endorses the conclusions of a 2013 paper that reaches a similar seemingly-implausible conclusion: http://andrewgelman.com/2013/08/14/the-robust-beauty-of-impr.... That lead me to a post by John Cook on the same topic: http://www.johndcook.com/blog/2013/03/05/robustness-of-equal..., where this paper was mentioned in the comments. The Dawes "The Robust Beauty of Improper Linear Models in Decision Making" paper is here: http://heatherlench.com/wp-content/uploads/2008/07/dawes2.pd...

The paper is interesting because it is a highly readable and extremely elementary introduction to the topic. Most HN readers don't even know the prehistory of the field, so I think it's useful to them.

The rudiments of modern theory can be understood with little more difficulty, and it is clear from those that any significant restriction of the number of degrees of freedom of a model reduces the chances of overfitting occurring, but also decreases the fraction of predicitions the model will get right, so the real issue is where exactly do we draw a line, and this is now understood quite well - the approach from the paper, for practical purposes, throws out the baby with the bathwater. The first few lectures of the course of machine learning by Yaser Abu-Mostafa are a really engaging introduction to those topics:

https://work.caltech.edu/

By the way Howard Wainer is a noted author of semi-popular (some formulas actually appear etc.) statistics books, so if enjoyed the writing, maybe a better use of time would be to read his newer and more general stuff:

http://www.amazon.com/Howard-Wainer/e/B000AP7SUU/ref=sr_ntt_...

It is interesting that an obviously "wrong" approach can have more success than expected. However, most people these days would use more modern forms of regularization (e.g. ridge, Lasso, elastic net, stagewise with early stopping). Modern optimization software can handle it.

As I noted in my other comment, equal-weight constraints are still used in one domain: neural network architectures (e.g. convolutional neural nets). This is not quite the same thing because neural nets are so highly parameterized, and the weights interact in so many nonlinear ways, that individual weights don't have any meaning independent of others. It does show us that the idea of equal-weight regression has still lived on in one specific place.

Equal weights are also used in creating ensemble averages. It's always bothered me that the IPCC predictions for climate are done by averaging the results across models of highly varying quality, with each one weighted equally. Instinctively, I'd think you'd want to put greater trust in the 'better quality' models, but perhaps equal weighting is more defensible than it appears.

This is extremely interesting. In the forecast domains I work, (I use neural networks on standardized inputs) I typically use 50-100 epochs at most for a few thousand rows (with some approx 10 input neurons) and I don't see networks improving very much after that.

However, our industry peers often use a few thousand up to several ten thousands of epochs, yet our software is regularly best or second best when compared by clients.

I think this may be the same effect: Network weights should be "ok" after a few dozen epochs and any training beyond that maybe wins a percent of variance explained but already risks overfitting.

We use the spare time to train more networks (on slightly different input data) whose results we aggregate after that.

Something else: can someone explain the "I don't make no nevermind", I suppose it's funny but I don't get it.

http://www.urbandictionary.com/define.php?term=it+don%27t+ma... though I don't get how it relates to the paper

Given the almost 40 years since this was written, how much has actually changed?

There have been discussions in a few bigger companies about including heuristics in decision making processes, sadly the human board members were unable to properly aggregate the pros and cons of such a move and the idea was dismissed.

Data set sizes have changed. That can make this issue (the tendency of OLS regression to overfit) less or more of a problem.

If you have 100 observations and 10 dimensions (N = 100, p = 10) then OLS can overfit, especially if the predictors are highly correlated or there's a lot of noise on the response. If you have N = 10^6, p = 10, then probably not. If you have N = 10^6 and p = 10^5, then overfitting is again a danger. If you have N = 10^5 and p = 10^6, then you can't even do an OLS regression because, when p > N, it involves inverting a singular matrix (and, to boot, a gigantic one). The rough guideline is:

    N/p > 100: very low danger of overfitting.
    N/p < 100, > 10: low danger of overfitting. 
    N/p < 10, > 1: high danger of overfitting. Use regularization (ridge, Lasso, etc.)
    N/p < 1: OLS not even well-defined. Regularization mandatory.

Equal weights is a primitive way of regularizing. You lose accuracy in the zero-noise case but get a model that is more likely to be "wrong" in a way that is harmless in prediction (because of correlations in predictors). These days, people are more likely to use ridge regression or, if they want a parsimonious model (few nonzero coefficients) they'll use Lasso or the elastic net.

Though you almost never see equal weights in a regression setting, you do see equal-weight constraints in neural network architectures (e.g. convolutional neural nets) to prevent overfitting. In neural nets, this usually can be done with absolutely no loss of accuracy because ANNs are already so highly parameterized that some form of regularization (weight decay, equal-weight constraints, early stopping) is mandatory.

I was asking about that in another thread. Do you have any references about you N/p rule of thumb? Thanks in advance!

That N/p rule of thumb is ultimately subjective, except for the fact that when N/p < 1, you can't use OLS anymore (because there is an infinitude of zero-training-error solutions). It depends on correlations of predictors and signal/noise ratio as well. Those are just approximate guidelines.

Reply on news.ycombinator.com