My experience with Kaggle's data-science competitions

Agreed, I think this made the front page solely riding off the hype of the recent AI and Machine Learning media surge.

I would love to hear more information though. Fun fact, part of my data analytics course next semester involves participating in Kaggle competitions.

OP here - thanks for the feedback. The post was intentionally designed to be quite light and introductory (specifically for those not familiar with Kaggle), but fear not, in the future I will be covering the practical aspects of data science (I'm presently working on a tutorial piece that explain how to use http://julialang.org - which is one of my favourite new languages for data science). Stay tuned.

<blog>I recently signed up for Kaggle. I think it's cool, too.</blog>

The data set used for the leaderboard is not the dataset used in the final evaluation. They also limit the number of final submissions you can have and the number of submissions you can make per day to prevent that.

Often the person at the top of the leaderboard will move down a few places in the final evaluation.

That seems quite high, though those people do need to make money somehow. The other businesses aren't really comparable because they have a much, much more money flowing through them and can afford to tax a smaller percent to stay competitive.

Thanks for sharing. Do you know whether that 50% fee is fixed, or is it variable based on the amount of prize money you put up? That does seem like a ridiculously high figure!

It is somewhat akin to spec work within the design field[1], in that only the top performers make money and everyone else goes home empty handed despite their hours of work. However, I've noticed that most of the near-winners tend to open source and share their solutions on GitHub, so one could argue that it helps the community (unlike say 99 designs, on which non winning entries are basically worthless).

[1]: http://www.nospec.com/faq

There are usually prizes, sometimes quite large. But some have accused them of making the industry worse by not hiring actual data scientists and outsourcing their work to people who are willing to do it pretty much for fun, and not all the prizes are enough to make up for the time and effort put into these competitions.

I sort of agree, but it's kind of ironic considering this is the field automating everyone else's jobs

Big Data has so many meanings. It really is a little absurd but I guess that is the life of a buzzword. My take on Big Data is 2 parts 1) Big Data is managing the large transactional, daily data of an organization. I believe this is what most people relate to Big Data i.e. map reduce, BI, etc. 2) Big Data is gathering bits of perceived dissimilar minutiae and finding meaningful information from it. This is classical statistics, engineering, and machine learning. The data itself may not be the scale of Amazon, Google or Facebook but it still computational intensive to develop algorithms for it.

Why? A multi giga dataset in machine learning is considered to be big enough to be called like that. An algorithm like support vector machine which complexity can be O(n^3) would be prohibitive in a single computer.