"Current" data can be fed in post-training.
Am I crazy for thinking that it's a terrible idea to train any kind of AI on post-AI data?
November 2022 is the LLM Trinity date.
Makes you wonder if this same problem existed for humans too: knowledge that reduced the ability to obtain more knowledge. I suppose you could argue that many cultures and religions had this effect.
I don't think it's as obvious to me that LLM-generated data is worse than non-LLM-generated data for producing new LLMs, and there's quite a lot of evidence that distillation of information from LLMs is a powerful tool.