Our perfect submission | Hacker News Reader

Our perfect submission

kaggle.com

96 pointsluu22 comments

Can someone please enlighten me on the context? Assume I know nothing about what Kaggle is, what the contest is about, and even what 'overfitting' means.

Kaggle is a website on which users can submit statistical models/programs/predictions into competitions, often involving rewards and many competitors.

The goal is usually to make interesting and insightful models based on a sample dataset. Overfitting occurs roughly when you tailor your model too specifically to the sample data. For instance, a regression with 10000 parameters might fit the data really well, but only because we included so many useless parameters.

A poorly designed contest on Kaggle had a submission that deliberately overfit.

If you're curious, there are many ways to address overfit models (look up model selection or AIC)

Kaggle hosts data science competitions. There is a public leaderboard and a private leaderboard. During the competition only the public leaderboard is visible. After the competition ends the private leaderboard is revealed and your standing on this private leaderboard decides your final score.

The public leaderboard gives some feedback on your model performance. But when a human is in the feedback loop, then there is a risk of overfitting. Overfitting can be explained basically as: "memorizing the data" or "learning from noise, not signal".

When you overfit, you do well in cross-validation and may do well on the public leaderboard, but your predictions do not generalize well to new data.

What this team did was to submit a lot of predictions and only take the predictions that improved public leaderboard score. Then they'd add slightly random noise and try to submit again. They repeated this until they ranked nr. 1 and left quite a few other competitors scratching their heads: How did they do this? Did they find a perfect ML algorithm? Is there data leakage? Are they cheating?

When the private leaderboard was revealed, this team dropped around 2000 spots. Their good performance on the public leaderboard was purely artificial. The contest was valid and well-organized (this could happen on any Kaggle competition with little data). They did not receive a prize.

There are other benefits to ranking well on the public leaderboard: It helps with teaming up with other high-ranking competitors and you can market yourself (recruiters are pretty interested in the top 10, eventhough the competition has not ended yet.)

I'll let the other answers stand for explaining Kaggle and their game, and try to answer "overfitting" with an ELI5 example.

Here is a hypothetical problem: Please write a calculator program, that takes input from stdin, containing one arithmetic expression per line (integers and + - etc), and simplifies the expression, returning one integer per line. Here's a sample input/output pair!

    Input
    2+2
    3+1*2

    Output
    4
    5

And you're all clever, and you write the following program:

    # python
    print "4\n5"

Congratulations, you pass my sample test case, but you have overfit your model to the sample data, and it will not generalize to other datasets for the same task.

If this were a Kaggle contest, and there were a leaderboard on how well people are doing on the sample data, you'd have a perfect score on the leaderboard. But then when it's time to see who wins the contest, and I try your program against some other inputs, you'd get 0%. Ha ha, funny joke you played!

If you enjoyed this, don't miss Moritz Hardt's entertaining blog post on Wacky Boosting (linked from the forum thread):

http://blog.mrtz.org/2015/03/09/competition.html

IPSC 2014 problem E looks like a machine learning problem, but the intended solution was to make multiple submissions and use the judge feedback to reverse-engineer the test data. The winning team needed only 6 submissions to hit 95% accuracy. It was one of my favourite problems from the contest.

http://ipsc.ksp.sk/2014/real/problems/e.html

What did they use the sixth submission for? You can win it in five submissions, can't you? :-)

On the 4th page towards the bottom, Kaggle says they will host the team's writeup "soon"; this was 26 days ago. I do not see it on the Kaggle blog [1]... still coming? Or did I miss it? (I scanned over the titles for the previous month and ran a search on BAYZ. Despite the topic I don't think the "Microsoft Malware Winners' Interview: 1st place, "NO to overfitting!"" post is related.)

[1]: http://blog.kaggle.com/

Usually the writeup is just supplemental to the posting's on the forum (and there is usually a 'post your solution' type deal where people walk through their process and sometimes post code) but I'm guessing it's just one of the competitions that will end up taking a month or so to gather all the interviews from the actual winners as well as BAYZ. Probably will not be too informative as they go quite into depth about the process and the linked blog post show's almost literal step by step of how to "game" kaggle leaderboards which is a huge aspect of competing (and why there is a private vs public dataset, why there are competition 'must enter by' deadlines and why people post benchmark code that will beat 25% of the current leaderboard before the competition is over and a lot of various other reasons).

Also looking at my submissions and notes from the competition (I stopped after the first week or so though) I even noted which 'groups' were most likely on the public leaderboard as it becomes easy to tell based on how your personal metric scores vs how that score results on the leaderboard since you know the evaluation metric for each competition.

Winning by overfitting. Is that the end for those contests?

Maybe they need a "any score lower than X will be considered as a bast score", or score in other aspects of the solution (like complexity)

Or just have a lot of data for scoring purposes.

They do have a lot of data for scoring purposes. It's the private data set that will be used for the final scoring.

No, they aren't going to win. The public leaderboard tests people's submission on toy data. Typical for contests like these. The real contest will be determined using realistic data, which is kept secret until after the contest is over.

Here is the public leaderboard for the now-completed contest, where they are in first place out of 2,257: https://www.kaggle.com/c/restaurant-revenue-prediction/leade...

Here is the private leaderboard, which I believe represents the actual score of the contest, where they are in place 1,962: https://www.kaggle.com/c/restaurant-revenue-prediction/leade...

That's, ah, not in the prize range, to put it lightly.

There's no problem with the contests here. It's just a particularly vivid, and entertaining to humans, demonstration of the well-known problem with overfitting.

If I am reading this correctly, the green numbers to the left of the teams in the private leaderboard represent that team's position relative to the public leaderboard. Note they're quite large... the 9th place team in the final scores jumped 1,058 positions! That's nearly half the field. If this is typical, and I have no idea (reply & let me know!), the public leaderboards are basically a joke.

So, why have 2 leaderboards? Or the private one has limited visibility by contestant?

(there seems to be simple ways for kaggle to solve this: put a delay on the score generation, limit number of submissions, limit precision of scores on the public leaderboard)

I'm guessing that the private one isn't visible to contestants or the public until the contest is over (hence "private").

The published dataset gives contestants an idea of how good they are, but prevents gaming the system by overfitting, because the contest is actually scored with the second unknown dataset.

Even with those other methods in place, the leaderboard would still favor methods that overfit. That is why the final score is determined on a dataset (the validation set) that is not used until the model is locked down. The public leaderboard is based on the test set.

A more in depth explanation on the use of three datasets for model building can be found here: http://stats.stackexchange.com/questions/19048/what-is-the-d...

The private one is revealed after the competition is over. Only the private leaderboard counts. To get an idea of how well your model performs, you get scores for, let's say 30% of the holdout data, on the public leaderboard. You can also use local cross-validation to gauge model performance.

There is a limit on the number of submissions. Usually 5 or less a day. Also precision of scores is visibly limited, but for final ranking all decimals count.

Still, this also confirms the obvious fact that the selection of the winning team based on that hidden, private test also gives no guarentee that it is not accidently overfitted, especially with tons of submissions. The only reliable option to solve it is to ask participants for self-contained programs and run it inside a CV loop on the organisers' hardware, but this seems too cumbersome to be implemented in reality; I think only TunedIT tries it, but without luck :L

It depends on the competition. For this competition it was to be expected as: There was little training data and there were a lot of outliers (the problem was predicting restaurant profits). Luck played a huge role here. The 9th place team likely had a mediocre highly-variant model that got a very lucky leaderboard split.

This forum post discusses competition variance and poses a metric to quantify "leaderboard shake-up": https://www.kaggle.com/c/liberty-mutual-fire-peril/forums/t/...

The Public Leaderboards are very helpful though! When you have setup a solid local cross-validation pipeline, and the public leaderboard agrees with your local evaluation, then you can try a lot more algorithms and parameters, without using any submission. Especially when working in teams this is important as you may have only 1 submission every 2 days.

Also, the more advanced Kagglers can use leaderboard feedback to increase model accuracy: Cluster the data sets with objective measures. Apply a modifier (restaurants from this region get 0.95 x previous prediction) and look at the result. If the split between public and private is random, and your clustering is objective, then an improvement on public leaderboard should reflect in private leaderboard.

This doesn't affect the actual contest results on the test set. Anyone who wants to overfit to the validation set can do so, getting a nice temporary leaderboard spot at the price of test set generalization, which is what determines the winners.

That said it'd obviously be nice to build leaderboards that are robust to this type of gaming. There's been some recent work, mostly driven by Moritz Hardt AFAIK (briefly referenced in the submission), using ideas from differential privacy to create leaderboards that leak only a bounded amount of information about the validation set: http://arxiv.org/abs/1502.04585.

I thought a perfect result on test data via overfitting was trivial (just memorize the data). What am I missing? (I've never done Kaggle and don't know how it's set up at all.)

You're missing the fact that the leaderboard dataset is not made available to the contestants, so they cannot memorize it.

The only way they can determine anything about the leaderboard dataset is by submitting a model, at which point they are told a single scalar score for that model.

Reply on news.ycombinator.com