Hacker News Reader

What you're talking about is not synthetic data. Synthetic data is data generated by an LLM. If the data is generated by a human, it's not synthetic.

You're talking about real data created and curated by humans to help in training LLMs.

View on HN

This is true and thank you for correcting me. I make a mistake of using synthetic data for what is human-generated data. There are augmentation tricks they could use on the human data but still human-generated data shouldn't be mistaken with synthetic data. I believe my core arguments still stand. Specifically that these models are learning their capabilities through specialized data that human professionals curate. There is no emergence or "general" intelligence. This is really what the open-weight models lack.

I agree with your premise though, we require specialized humans to generate the highest quality data to improve the models, and it would be more impressive if the models could do this themselves...

Anthropic have done experiments with having "dumber" models try and make a smarter model, smarter, but it still seems tricky so far.

But if you think about humans, we have the same issue. If you have a super smart person, how do they get smarter? Can you have dumber people make a smarter person even smarter? It's a hard question.

Reply on news.ycombinator.com