Hacker News Reader

Comment by jph00 | Hacker News Reader

I'm the Kaggle President & Chief Scientist that is interviewed in this article. Feel free to ask me any questions that you have about the role of domain experts in data -driven decision making (or any other relevant topic!)

(It didn't occur to me to submit the article to HN - silly me! Thanks ivoflipse for doing so.)

View on HN

If I create 1000 predictive models, with random parameter settings, some of those models will outperform others. In fact, the accuracy rankings of the models given a single hold-out validation set will form their own distribution. It is difficult, if not impossible, to tell if one model outranks another b/c it is actually better or just occupies a different place in that distribution due to the random properties of the hold-out set.

So, why do you believe this isn't happening when hundreds of people submit their models to be scored against a secret hold-out set? Is there any evidence that the hundreds of entries are forming a better distribution of accuracy scores than random models? Is there evidence the winners of these competitions truly have better models and don't just have a luckier placement in a random distribution of accuracy scores?

Yes, we can calculate the distribution and find the likelihood that a given team would beat another if the data is resampled. Here's a detailed example: http://www.tiberius.biz/pakdd07.html

The size of improvements seen in competitions vs previous best performing models (eg Allstate competition showed improvement of 270%) is well beyond that explicable by chance. Also: the test set size in most competitions is hundreds of thousands of rows - sampling error is very small in these cases.

I still find it hard to believe the people in the top 10 aren't all at the theoretical limit of predictive accuracy with their final order being determined by nuances in the experimental design (things like how the raw data was collected).

Which would be the sort of thing statisticians live with all the time, except that Kaggle then makes it winner take all.

That certainly does happen - indeed our tag line is "making data science a sport"... Like in all sports, there can be a stochastic element to who ends up winning!

However, in private competitions (which is where the money is!) everyone selected to participate wins something. It's more like golf - make the cut, and you'll definitely make a few dollars, but the winner will take home the biggest prize.

Do you have a sense of how much computational resources matter? I remember reading an article on "overkill analytics." Do you see any victories by amateurs who don't have access to particularly powerful clusters?

Good question. Most winners use only laptops with R or Python (Matlab is also popular).

Feature engineering and common sense is definitely more important than computational resources.

Do the same few people tend to win most of the competitions? I would assume the number of contest victories would follow Ziph's Law, with a sharp decline after the top few. In which case, it might be that there are a few really good general data analysts rather than the topical experts being poor at their jobs.

Yes, that is what we see.

Reply on news.ycombinator.com