> A lot of the corpus feeding GPT-3 was web scraping and now that source is tainted for future GPT-x models
It might be possible to filter out GPT-3 generated text from future training data. Simply feed part of the text to GPT-3, and if it is way too good at predicting what follows you throw it out. The same trick could be used to detect students writing essays with it.
This trick will stop working though as more variants of good text prediction algorithms appear, unless we can do the same test against each one.