I thought synthetic data is what is partially training the new multimodal large models, i.e. AlphaGeometry, o1, etc.
I don't see how you can "bootstrap a smarter model" based on synthetic data from a previous-gen model this way. You may as well well just train your new model on the original training data.
The Information in the data isn't just about the output but its rate of occurrence/distribution. If what your base model has learnt is only enough to have the occasional flash of brilliance say 1 out of 40 responses and you are able to filter out these responses and generate as much as you like then you can very much 'bootstrap a better model' by training on these filtered results. You are only getting a function of the model's parameters if you train on its unfiltered, unaltered output.
E.g. you can't just synthetically generate code, something or someone needs to run it and see if it performs the functions you actually asked of it.
You need to feed the LLM output into some kind of formal verification system, and only then add it back to the synthetic training dataset.
Here, for example - dumb recursive training causes model collapse:
I suspect most foundational models are now knowingly trained on at least some synthetic data.