Hacker News Reader

Comment by nostrademons | Hacker News Reader

The two of you are asserting different hypotheses than the OP presented: diminishing returns != decreasing returns. The Google paper found that increased amounts of data always improved performance, but did so at a lower rate the more data that had already been provided. First derivatives vs. second.

The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.

View on HN

I'm only talking about the Google paper brought up in this thread, because logarithmic growth does asymptotically decrease over time. I didn't say that the Google paper asserts absolute decreases over time, such that more training data actually makes results worse. Just that more data becomes too expensive to be worth it in a logarithmic context.

The headline isn't consistent with the article itself, which states (somewhat confusingly):

* As the number of product data grows, the benefits were negligible

* More observations per product was important

* The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns.

In any case, this seems to be a case of "diminishing returns."

>> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.

Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".

Few big tech companies pay for training data - it usually arises organically out of usage data from their products. You need to collect this anyway for product function / business metrics / abuse prevention, and build the cleaning & processing pipelines. So the only marginal cost of feeding more training data into your machine learning pipeline is the computational cost of training it, which is usually tiny fractions of a penny per sample.

If it was so simple to collect, process and train with (very) large amounts of data, everyone woudl be doing it. Instead, it's just a few very large companies that can do that, Google, Facebook et al.

Anyway, the cost per example doesn't have to be astronomical. If you need a few millinos of those, you can pay a fraction of a penny and still have a big black hole in your budget, unless you can significantly improve performance.

Only a few big companies can do it because there's a bootstrapping problem. To get large amounts of virtually free data, you need lots of users who have signed up for giving you their data in exchange for a useful service. This was much easier to achieve for companies started between 1995-2005, when the web was young, because the Internet was such a huge leap forwards over what came before it. Existing startups now have to compete with the products of these giants, many of which have been enhanced by years of machine learning. That's challenging.

To give you a sense of how cheap computing power is, my startup regularly processes roughly 2B webpages with some complicated algorithms that need to go node-by-node over the whole DOM tree. That's roughly 77TB of (gzipped) data, and around 100 trillion nodes. It costs me a few hundred bucks of AWS time. That's a rounding error for a big corp; a single data scientist's salary for one day will run you around that much.

Reply on news.ycombinator.com