Hacker News Reader

Scaling laws are largely about data and parameters, compute mostly more indirectly and weakly.

If there's no extra data most scaling predictions start to look bad.

None of this is surprising. Every AI/ML researcher has done a back of the envelope estimate to see how much data there is and when this must stop. Me included. It's a common topic at conferences.

Am I understanding it correctly that the gist of your argument is that OpenAI, Google etc have run out of available data to train LLMs?

What's your estimate for how many tokens that represents?

I suspect it’s more the end of an avenue than the end of the road, but it sure seems like throwing more data and compute is hitting diminishing returns on existing methods.

It was never a "law", it was what companies selling models made their investors believe but the limits were obvious from the start.

Is the obstacle compute, or is it available training data?

At some point the LLMs are just redigesting their own vomit.

Very likely the training data.

Punctuated equilibrium