Hmm, forgetting natural language for a moment and instead considering programming languages: it’s pretty easy to generate nonsense but semi-plausible looking ASTs without the help of AI. Could this be used to attack GitHub’s copilot?

Step 1. Release a tool that generates nonsense code across a thousand repositories, and allow anybody to publish crap to GitHub.

Step 2. Copilot trains on those nonsensical repositories because it can’t distinguish them from the real thing.

Step 3. Copilot begins to generate similar crap.