That's correct. I saw a paper recently that showed how LLMs performance collapses when they are trained on synthetic data.