It's possible that would be a very slow, essentially "serial" process--adversaries don't really find a new trick until the AI can avoid the old, like with the pass attack in this paper. I'd also be curious whether training against adversaries makes the networks better or worse at general-purpose play or causes no change, but there's an argument that more-robust-but-slightly-worse-on-average is an overall improvement.
It also seems relevant to looking at weird behaviors of other models, like what we're seeing as people poke at LLMs more. Some of what we're seeing might simply be that they respond weirdly to some fraction of all prompts because they're not quite baked; also, we haven't defined their goals all that well yet. But automatic searches for problematic classes of input seem like a possible tool for keeping things more on the rails.