If you make a mistake in applying those fixes, you will have a security hole. When you spot that hole you can close it up and now you are back to 100% protection.
You can't get that from defenses that use AI models trained on examples.
To me, that's a liberating thought: we tend to operate under the assumptions of SQL and the DOM, that there's a "right" solution that will allow those full mappings. When we can't see one for LLMs, we sometimes leap to the conclusion that LLMs are unworkable. But allowing the full map is a constraint we can relax!
Is there potentially a way to implement out-of-band signaling in the LLM world, just as we have in telephones (i.e. to prevent phreaking) and SQL (i.e. to prevent SQL injection)? Is there any active research in this area?
We've built ways to demarcate memory as executable or not to effectively transform something in-band (RAM storing instructions and data) to out of band. Could we not do the same with LLMs?
We've got a start by separating the system prompt and the user prompt. Is there another step further we could go that would treat the "unsafe" data differently than the safe data, in a very similar way that we do with SQL queries?
If this isn't an active area of research, I'd bet there's a lot of money to be made waiting to see who gets into it first and starts making successful demos…
[1] check out Robert Miles' excellent AI safety channel on youtube: https://www.youtube.com/@RobertMilesAI