Multi-modal prompt injection image attacks against GPT-4V

It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing.

Instead, the model itself has been trained on both text and images at the same time.

So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in the prompt.

I'd love to understand this more. Strings of text get tokenized into a sequence of integer tokens - is there a similar tokenization step for image inputs? What does that look like?

Even if development were to freeze, they didn't improve at all from this day onwards, and their many monumental flaws (prompt injection, hallucination, inability to reason etc) were never solved, I still think they'd be worth studying and using.

I expect we could still spend years figuring out new capabilities in the models we already have access to today.

In addition, the fact that Simon is very bullish about LLMs makes his writing on LLM attacks more effective, because a decent chunk of people warning about LLM attacks are also skeptical about the long-term viability of the technology or think that it's over-hyped, so LLM fans tend to to dismiss them as being "anti-AI" and say that they're playing up the dangers. Nobody can make that claim about Simon.

For whatever reason Simon's writing has consistently attracted a lot more attention than other researcher releases about LLM attacks; and what has he done with that popularity and outsized attention? He's raised awareness about the issue and pointed back to some of that research!

I just don't think the criticism in this case is warranted, I think Simon's writing here is worthwhile regardless of how anyone feels about the potential of LLMs. Whether they're the future or not, these vulnerabilities exist today in products that are being rolled out today. That's worth taking about if LLMs aren't the future, but it's even more worth talking about if LLMs are the future.

In my opinion there exists a rather simple semi-solution (and no, I do not claim to have cracked prompt injection! :-) ): the user is able to mark parts of his input as "trusted" or "untrusted", and the AI is implemented to handle data this way.

For example, the user would mark his prompt (that he thoroughly thought through) as "trusted", but the image which provides additional information, or is the object to run the prompt on, as "untrusted".

This is related to established concepts in IT security:

- W^X: memory pages are either writable or executable, but not both (of course the term "W^X" is a little bit confusing since if W=0 and X=0, W^X is 0, but is an allowed configuration) - corresponding to untrusted or trusted parts of the prompt

- sandboxing of the virtual machine in browsers that execute untrusted data from the web (untrusted part)

I imagined simply tagging the tokens as command and data, ie say instead of just (43) as a token, it's (43, 1) for command and (43, 0) for data. Then ensure through adverserial training that whatever interprets say images does not spit out command tokens.

edit: or better yet, just have it spit ut plain tokens like (43) and ensure they're data tokens by manually converting them to data tokens (43, 0).

Isn't this simple? Have two different tokenizer with non-overlapping output ranges, one for “trusted commands” and one for “user input”, do training with various mixes of commands and input with the appropriate reactions to each, and then have the trusted command fixed (or selectable from a small set of preselected options/payterns based on interaction style [chat might used trusted tokens to indicate which participant each piece of the exchange is from, as well as for the internal system prompt, for instance]) for external users and allow them to freely specify only input that gets sent to the “untrusted input” tokenizer. Even if you allow pretokenized input, you can sanitize it to accept only tokens from the untrusted space.