The reason why LLMs cannot be purely* pre-trained on LLM outputs is because LLM output diversity is low.
This implies two things:
1) There is going be no functional difference between hypothetical Qwen 4.5 and Fable 5.5
2) LLMs are not AGI
- https://arxiv.org/abs/2609.37891
- https://arxiv.org/pdf/2508.10975
- https://www.datologyai.com/blog/eating-the-rainbow-a-primer-...
As well as recent model releases that used large fractions of synthetic data (Arcee Trinity):
- https://arxiv.org/pdf/2602.17004v1
It's mostly a matter of increasingly complex and sophisticated sampling methodologies to sift for the valuable synthetic tokens.
There have also been some interesting results about models that use the same data converging to very similar behavior, but I can't find the specific reference at the moment. That one tracks intuitively with the notion that the model is building a latent representation of everything it sees.
Upon a quick glance of the data sections of those papers, every data set you have there is human data augmented by LLMs. Also not to mention the implicit data curation and generation by humans.
Can a model be trained purely on LLM-Outputs and access to tools such as a Turing complete environment?
No one paraphrased the laws of gravitation and handed it to Newton...
There are some things that are true (I hope), so using those concepts, approximated by a human text filter, as seed seems reasonable. Hopefully the diversity in sampling methodologies and opinions about what makes a meaty token pays some dividends down the line for model diversity.
And with ever-more LLM usage both in labs and the real world, the task of finding new/“creative” out-of-average-distribution material will only continue to increase in difficulty. Meaning if this continues, in the future, all output becomes the equivalent of the color brown?
But I should’ve clarified, I meant the scraping aspect of it, since more and more content on the open web will be LLM output.