Hacker News Reader

Comment by crazygringo | Hacker News Reader

Exactly. This is neither a new idea, nor is it foolproof in the way that SQL sanitization is.

I suspect that at some point in the near future, an LLM architecture will emerge that uses separate sets of tokens for prompt text and regular text, or some similar technique, that will prevent prompt injection. A separate "command voice" and "content voice". Until then, the best we can do is hacks like this that make prompt injection harder but can never get rid of it entirely.

View on HN

It's 175 billion numeric weights spitting out text. Unclear to me how we'll ever control it enough to trust it with sensitive data or access.

The number of weights is irrelevant. It's about making it part of the architecture+training -- can one part of the model access another part or not. Using a totally separate set of tokens that user input can't use is one potential idea, I'm sure there are others.

There's zero reason to believe it's fundamentally unsolvable or something. Will we come up with a solution in 6 months or 6 years -- that's harder to say.

The number of weights(unless extremely small) is irrelevant but the general idea is not. You can't train a neural network on internet scale data and expect to control what it can say.

We can train but we don't teach them anything. They learn from data directly and we don't know or understand what they learn so we can't adjust what they learn directly.

You can't make "always obey these types of tokens" a part of the architecture or training. It's a concept that doesn't even make sense for the vast majority of text it pre-trains on.

"Solving" prompt injection is solving alignment. It's not happening.

There are good reasons to think it’s fundamentally unsolvable within the LLM architecture. The reason LLMs are good at following instructions is because they have an enormous corpus of data. That corpus powers both the comprehension of inputs and the comstruction of outputs. Don’t forget that it’s a token predictor at the bottom! If the instructions are separate from the data, then all of that power goes away.

My point isn't the number of weights, it's that the whole model is a bunch of numbers. There's no access control within the model because it's one function of text -> model weights -> text.

It's like in-band signalling or out-of-band signalling in telephony. With in-band signalling, you could use a blue box and get calls for free.

Sql sanitization isn't foolproof either. That is why prepared statements are the best practice.

SQL sanitation is foolproof in the sense of it being possible to do 100% right. We don't do it much because there are other options (like prepared statements) that are easier to get 100% right.

This is an entirely different thing from trying to reduce the probability of an attack working.

Everything is in theory possible to do 100% right. The difficulty of doing so is why people choose better solutions, like prepared statements.

The only part that isn't foolproof is remembering to do it. If you run the sanitization function, it will work.

Unless you're using a build of msyql that predates mysql_real_escape_string, because the _real version takes the connection character set into account and the previous version didn't.

Reply on news.ycombinator.com