Kaggle Post-Mortem: The dangers of overfitting
gregorypark.org
gregorypark.org
Black box machine learning is unlikely to improve a meaningful metric beyond a stock SVM or random forest. It's possible to tune an algorithm to achieve arbitrarily good training set performance, of which you can include the small "test set" evaluation, but this does not make a generalizable, practical tool.
Opportunities might exist if you have the opportunity to feature engineer with domain expertise in order to keep improving your Bayes error. Kaggle is designed not to allow for this, though.
I took part in the last essay scoring competition on Kaggle.
In that competition, the data set was the original text of the essays, scored by human raters. A lot of the effort in our submission was in developing features. We didn't spend much time on our entry, but we did use domain specific features (e.g. writing code to calculate http://en.wikipedia.org/wiki/Gunning_fog_index), implementing spell checking, etc.
That entire competition took place within the Kaggle framework.
Furthermore, in addition to the training-set (which we did most of our development with, using cross validation) and the test-set, on which the final scores were calculated, there was also a validation-set, from which the public scoreboard was calculated. We could only submit our predictions for the validation set twice a day, and were told only our overall accuracy on the set -- not given the scores of each essay.
The role of the validation set in Kaggles framework was surely to prevent errors such as overfitting. In fact, we had a situation where our cross-validation accuracy increased, and validation-set accuracy decreased, because of a bug where our feature selection didn't properly implement cross-validation (similar to the linked article).
The difference between the cross-validation and validation-set scores drew our attention to this problem.
That all happened within the Kaggle framework.
So I don't really understand criticisms as general as 'Kaggle is doing ML wrong'?
Here by validation score I mean the combination of both your own cross-validation procedure on the training data, and the validation score that kaggle calculates and displays on the leaderboard while the competition is running.
I imagine if the essay scoring competition had been extended to run for another 10 years before the final scores on held-back test data were reported we would see a similar trend of overfitting in most if not all of the entries.
edit: I might be underestimating the sophistication of your approach or the approaches of others.
At least in the Kaggle competitions I have personally entered, I have used my own cross-validation scores and the Kaggle leaderboard scores to tune my approaches, without properly taking into account that this breaks the validity of the subsequent validation procedures, since I have no way of validating my tuning.
But thats hardly a criticism of Kaggle.
Its the responsibility of the person building the ML model to take steps to avoid overfitting. Even if competitors fail to take these steps, the fact that Kaggle has a built in validation-set helps avoid overfitting.
Yes, with enough time it would still be possible to overfit, but I don't see how that is Kaggle's fault.
If anything, they've a framework to discourage overfitting, not encourage it.
So, lets say you have a large training set - a large number of randomly chosen example essays, and a score for each essay. You want to develop a model that will predict/guess scores of other essays, in future.
Your model is going to be evaluated against a set of essays - the test set - which you do not see until after the competition is over (at which point you may no longer make changes to your model).
Now, in an ideal world, we'd be able to take just one look at the training set, and instantly learn the best way there is to predict similar essay scores in future. But, in practice, that's not going to be the case. We, the designers of the ML approach, initially don't know how to build essay grading systems. So we are going to need to do a development process, where we think, then try a particular approach, then see how well it works. And we have to iterate on that process.
So, the key bit there, is that we need to see whether a particular approach is working. We can't check the test-set - we don't get access to that until the end of the competition. So, instead, we take the training set, and partition it up into 2 parts, lets call them A and B. Essays from both A and B are in the training set, as we've just said, hence we know the scores for the essays of both A and B.
We now train our model with essays from A, without it being allowed look at essays from B. Then, afterwards, we have it make predictions/guesses of the scores of essays in B, and see how well it does.
This allows us to evaluate how well our modelling approach is doing.
In practice, we repeat this procedure for many different partitions of the training set, into many different A and Bs.
So, thats cross validation.
The key thing to realize is that if we use any information from B, when training A, its not really a fair evaluation. The model we make from A will have been contaminated with information from B.
Which is simple enough.
The thing is, in practice, before doing any of this, people generally go through a conceptually separate process, of deciding what features their model is going to use - 'Feature Selection'. Some part of this process is intuitive, but there are also algorithms people use, to help them do feature selection.
The wrong way to do Feature Selection, is to run your Feature Selection algorithms on the entire training set, and then afterwards train your model on one partition A of the training set, and test it on separate essays from B.
This is wrong, because the essays from B were seen in the feature selection process. Sometimes that doesn't matter - if you are dealing with large data sizes, etc. But on the Kaggle essay competition, in particular because we had rather a large amount of features (e.g. bag-of-words n-grams, parts-of-speech n-grams), relative to the number of training examples, it certainly did matter.
As a result, we saw much higher scores on the training set, than we would have gotten on the test set.
The right way to do things is to do any automatic feature selection only on the training partition of the training set, A. One way to do this is to build any algorithmic feature selection into your process after the point you partition into A and B, during cross validation. But its very easy, when hacking on a competition solution, over a weekend, to accidentally put things in the wrong order, in the codebase.
For this reason, if you have limited training data, its generally a good idea, at the very start of the process, to take some training data, and put it somewhere completely separate, until you think your algorithm is ready to go, and then use it as a final test. People often call this the validation-set.
"Kaggle is doing ML wrong. It has the opportunity to teach some people about these perils": We have made several (reasonably successful) attempts to encourage participants to use cross validation (including a competition called "Don't Overfit: http://www.kaggle.com/c/overfitting). The incidence of over fitting has decreased dramatically since the site first launched - but it hasn't been eradicated. If people have ideas about how we can do better at educating our community, we're all ears. (Unsurprisingly, once someone makes the over fitting mistake once, they never make it again.)
As an aside, one of the key ways we try and promote education is the "How I did it" posts on the Kaggle blog (blog.kaggle.com).
"Black box machine learning": Actually our number one learning on ML best practice is that for a competition with rich data (like the essay scoring competition mentioned elsewhere in the comments), feature engineering (sometimes known as variable creation and variable selection) is WAY more important than whether or not one uses SVM or random forest.
So modest.
(Sigh. The point of my joking remark is not to dump on Anthony Goldbloom, it's that he's slightly more involved than "working there". He founded Kaggle).
I'm referring to the blue lines in the scatterplots of the rank vs. no of submissions: those unidentified curves should have been straight lines. What's the point of having some high degree polynomial there?
For those not in the knowing, this book proves how most of our reasoning is false most of the time, shows that people vote based on facial features, that success is luck, that the world is out of our control, that experts are worse forecasters than monkeys randomly picking options, and the better the experts the worse the forecast. Depressing.
Edit: one thing reassuring is that it explain rationally how random people moderately clever and sometime plain despicable can become the head of very successful companies.
> that experts are worse forecasters than monkeys randomly picking options
For a more nuanced study on this, see Expert Political Judgement: How Good Is It? How Can We Know? by Phillip Tetlock.
If you aggregate expert opinions then yes, they barely beat monkeys. Happily they outperform aggregated non-expert opinions.
However you can classify how experts reach conclusions and find that some categories of thinkers (foxes) will consistently outperform other categories (hedgehogs).
Mind you -- humans and monkeys both are absolutely trounced by statistical models.
In the expert prediction study, experts who had a universal framework which they extended rigidly to every problem performed poorly on both calibration (correlation of confidence and actual post-factual probabilities) and discrimination (correlation of more/less/same predictions to actual outcomes).
Foxes tended to throw together a bunch of thoughts, including from incompatible theories, express a lot of "one of the one hand, and on the other" sort of thinking and qualified most of their predictions. They did much better on calibration and somewhat better on discrimination.
Basically, Hedgehogs tend to make more extreme predictions ("Russia will fall into civil war by 1997") more confidently ("it's almost certain") than Foxes. But statistically, about 40% of the time variables in the complex systems being forecast actually stayed within a band and neither decisively moved up or down.
I know that you're probably thinking of holes and objections to the study already -- read the book. Tetlock and his collaborators were astonishingly thorough in trying to deal with all the relevant arguments and counterarguments with actual data.
Quite the contrary. Your description of the differences is clear and the outcomes from the study fit well with a similar meme in machine learning around ensemble methods.
Perhaps they might be onto something with "kaggle prospect", but unless they pivot in some creative new direction, it's hard to see the service being very useful.
We're actually using public competition as a way to find out who are some of the world's strongest data scientists. Once somebody has performed well in several public competitions, we start inviting them to private competitions where: 1. we invite 15 members; 2. prize money is generally much higher; and 3. everyone wins something (the higher the position the more one wins)
We're aiming to give many of the world's data scientists the opportunity to earn great full time incomes by competing in our competitions.
In other words, more Snake Oil from the computer industry. In some (most?) cases, the salesmen may even believe that the Oil works. That is, they may not be acting fraudulently, but just foolishly (as are their clients who believe the hype).
Overconfidence in math and computers to solve problems that are not suited to be solved this way. In some ways this is partly responsible for the global financial mess. Reliance on models that quants and their bosses would defend vigorously, because they work to make money for the bosses (because clients believe the hype), but which are, overall, in the long run, not sound.
Kaggle has taken an important niche and created a market for buyers and sellers. I was skeptical when I first saw it mentioned but then, I'm usually amazed when anything succeeds. :D
Science and math have varying levels of rigor depending on the domain. On the other hand, not using them has a consistent outcome: guessing and ignorance.