The biggest risk to that security risk is its own name. Needs rebranding asap.
But I'm at least happy to jump to other terminology if that changes, I do think that calling it "prompt injection" confuses people.
I think I remember there being some effort a while back to build a more extensive classification of LLM vulnerabilities that could be used for vulnerability reporting/triaging, but I don't know what the finished project ended up being or what the full details were.
"Injection" is a narrow and irrelevant definition. Natural language does not follow a bounded syntax, and injection of words is only one way to "break" the LLM. Buffer overflow works just as well-- smalltalk it to death, until the context outweighs the system prompt. Use lots of innuendo and ambiguous verbiage. After enough discussion of cork soakers and coke sackers you can get LLMs to alliterate about anything. There's nothing injected there, it's just a conversation that went a direction you didn't want to support.
In meatspace, if you go to a bank and start up with elaborate stories about your in-laws until the teller forgets what you came in for, or confuse the shit out of her by prefacing everything you say with "today is opposite day," or flash a fake badge and say you're Detective Columbo and everybody needs to evacuate the building, you've successfully managed to get the teller to break protocol. Yet when we do it to LLMs, we give it the woo-woo euphemism "jailbreaking" as though all life descended from iPhones.
When the only tool in your box is a computer, every problem is couched in software. It smells like we're trying to redefine manipulation, which does little to help anybody. These same abuses of perception have been employed by and against us for thousands of years already under the names of statecraft, spycraft and stagecraft.
Jailbreaking is more akin to social engineering - it's when you try and convince the model to do something it's "not supposed" to do.
Prompt injection is a related but different thing. It's when you take a prompt from a developer - "Translate the following from English to French:" - and then concatenate on a string of untrusted text from a user.
That's why it's called "prompt injection" - it's analogous to SQL injection, which was caused by the same mistake, concatenating together trusted instructions with untrusted input.
The problem is that SQL injection has an easy fix: you can use parameterized queries, or correctly escape the untrusted content.
When I coined "prompt injection" I assumed the fix would look the same. 14 months later it's abundantly clear that implementing an equivalent of those fixes for LLMs is difficult to the point of maybe being impossible, at least against current transformer-based architectures.
This means the name "prompt injection" may de-emphasize the scale of the threat!