If the output of prompts can be improved with "think step by step", "tree of thoughts", etc., one can produce high quality training data with the LLM itself which the next iteration of that model can use. Rinse and repeat.
The interesting piece is if we can get LLMs to reason well, that finite reward capture can take LLMs much farther than possibly humans could do with the same signal.
Similarly to when a machine has an efficiency of 1%, optimizing it to 10% does not violate thermodynamics.
There is a limit to what you can learn from limited knowledge. Every bit of information you learn can only divide the space of possible theories by half. However, current LLMs are ludicrously far from that limit.
And if you _do_ let LLMs look at the world, and train on that, they also wouldn't have that limitation.
it may be higher quality, but far from perfection.
After many rinses and repeats dataset will accumulate significant amount of errors.