Everyone who's thinking about the ramifications of prompt injection attacks now, please consider: This is really just a specific instance of the AI alignment problem. What about when the AI gets really smart, and tries to achieve certain goals in the world that are not what we want? How do make sure that these soon-to-be omnipresent models don't go off the rails when they have the power to make really big changes in the world?

It's the same problem. We have no way to make an AI that's 100% resistant to prompt attacks, OR 100% guaranteed to not try to act in a way that will result in harm to humans. With our current approaches, can only try and train it by bonking it on the nose when it doesn't do what we want, but we don't have control over what it learns, or know whether it has correctly internalized what we want. Like the article says, if you've solved this, that's a huge discovery. With the current intelligence level of GPT it's just a security hole. Once AI's become smarter, it's really dangerous.

If you weren't worried about prompt attacks before and are now, I would say that it makes sense to also reconsider whether you should worry more about the danger of misaligned AI. That 1% or 0.01% situation is guaranteed to come up sometime.