wunderwuzzi23

>> Are you suggesting people will plug open ended APIs that allow the bots to charge any amount without validations?

Certainly. A good example (not an Orderbot, but real world exploit) was "Chat with Code" Plugin, where ChatGPT was given full access to the Github API (which allowed to do many other things then reading code):

https://embracethered.com/blog/posts/2023/chatgpt-chat-with-...

If there are backend APIs, there will be an API to change a price or overwrite a price for a promotion and maybe the Orderbot will just get the context of a Swagger file (or other API documentation) and then know how to call APIs. I'm not saying every LLM driven Orderbot will have this problem, but it will be something to look for during security reviews and pentests.

A real Orderbot has the menu items and prices as part of the chat context. So an attacker can just overwrite them.

During my Ekoparty presentation about prompt injections, I talked about Orderbot Item-On-Sale Injection: https://youtu.be/ADHAokjniE4?t=927

We will see these kind of attacks in real world applications more often going forward - and I'm sure some ambitious company will have a bot complete orders at one point.

Very true. If you are curious I have an entire collection of such prompt injection to data exfiltration issues compiled over the last year. From Bing Chat, Claude, GCP, Azure they all had this problem upon release - and they all fixed it.

However, most notable though is that ChatGPT still to this day has not fixed it!

Here is a list of posts showcasing various mitigation and fixes companies implemented. Best is to not render hyperlinks/images or use a Content-Security-Policy to not connect to arbitrary domains.

https://embracethered.com/blog/tags/ai-injections/

It's actually slightly worse, because it is forced sharing! The recipient doesn't have to accept an invite -someone can just share a google doc with you and it will be visible in your drive. It's like sending someone an email (which is another attack vector this could have been triggered)

Don't forget about ROPC!!

There is also an MFA bypass that I see often, it's the Resource Owner Password Credentials (ROPC) flow in OAuth.

Especially when you have an Microsoft M365/Azure tenant. Pretty much every client that I have ever tested had this issue. When ROPC is configured (which is the default) then you can just use a simple password to logon (and it bypasses MFA).

More details and a tool to test your own tenant: https://embracethered.com/blog/posts/2022/ropci-so-you-think...

If you use M365/Azure test and see if you can logon without MFA, you might be surprised.

Great to see this getting more traction.

Two things I wanted to add:

1) The image markdown data exfil was disclosed to OpenAI in April this year, but still no fix. It impacts all areas of ChatGPT (e.g. browsing, plugins, code interpreter - beta features) and now image analysis (a default feature). Other vendors have fixed this attack vector via stricter Content-Security-Policy (e.g Bing Chat) or not rendering image markdown.

2) Image based injection work across models, e.g. also applies to Bard and Bing Chat. There was a brief discussion on here in July about it (https://news.ycombinator.com/item?id=36718721) about a first demo.

I'm very glad that you are focusing and helping educate the industry (both from pure tech but also security aspects). And agreed that as a daily user of LLMs myself, the potential (and practical use cases already) are enormous. E.g. Writing code and rapid prototyping will never be the same for me.

By the way the last official answer I got from OpenAI regarding the markdown issue was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution.

My current takeaway is that with LLMs, the assumption always needs to be that the LLM is capable of outputting any content, and the client needs to safely (and securely) handle whatever is returned.

Btw. I show cased these kind of image based prompt injections in July (with Bard and Bing Chat, which presumably already used OpenAI's tech), and the example I created (the robot slipping on a banana) was shown during Blackhat in Las Vegas this year. Tweets: https://twitter.com/wunderwuzzi23/status/1681520761146834946

Thanks for the shout out, the last answer I got from OpenAI in that regards was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution.

My conclusion is that with LLMs, the assumption always needs to be that the LLM might be capable of outputting any content, and the client needs to safely handle whatever is returned.

The vulnerability is the automatic insecure rendering of image markdown. One way to trigger it is with an indirect prompt injection payload. The scenario is that the user analyzes some text/data, which contains malicious instructions. The owner of the text doesn't have access to the chat history (it's just some random text somewhere), it could be a comment on a webpage, text inside a pdf file, copy/pasting, or even instructions hidden inside an image the user analyzes and sends to the LLM. You can find many examples of indirect prompt injections on my blog (e.g. analyzing YouTube transcripts,...). Just yesterday I put up a video explaining the various TTPs (and also fixes companies put in place): https://www.youtube.com/watch?v=L_1plTXF-FE Hope that helps.

Wonder if for the Enterprise version they will fix the Image Markdown Data Exfiltration vulnerability that's been known for a while.

https://embracethered.com/blog/posts/2023/chatgpt-webpilot-d...

Seems like a no-go for companies if an attacker can steal stuff.

After I shared some POC exploits with Plugins OpenAI added this requirement it seems.

However as far as I can tell, and most recent testing shows, this requirement is not enforced: https://embracethered.com/blog/posts/2023/chatgpt-plugin-vul...

I'm still hoping that OpenAI will fix this at the platform level, so that not every Plugin developer has to do this themselves.

It took 15+ years to get same-site cookies - let's see if the we can do better in here...

This was added recently, and is not enforced as far as I can tell.

See my blog post: https://embracethered.com/blog/posts/2023/chatgpt-plugin-vul...

Yeah, for that reason it can probably do many things that aren't actually intended when you want to discuss your code with ChatGPT, like it can make private repos public and things along those lines...

https://embracethered.com/blog/posts/2023/chatgpt-plugin-vul...

During an Indirect Prompt Injection Attack, an adversary can also force the creation of issues in private repos and things along those lines.

I wrote about some of these problem in the past e.g. see: https://embracethered.com/blog/posts/2023/chatgpt-plugin-vul... on how an attacker might steal your code.

Some other related posts about ChatGPT plugin vulnerabilities and exploits:

https://embracethered.com/blog/posts/2023/chatgpt-cross-plug...

https://embracethered.com/blog/posts/2023/chatgpt-webpilot-d...

Its not very transparent when and why a certain plugin gets invoked and what data is sent to it. One can only inspect afterwards basically.

Embrace The Red: https://embracethered.com

Blogging mostly security related things, red teaming, pen testing, AI security, threat modeling and stuff.

Yes, there are multiple ways: Sending data to a plugin or having the indirect prompt injection emit a markdown image. I reported it to OpenAI on April, 9th also provided a CVSS score and argued image injection would be high severity issue once built-in plugins are available. After some back and forth, I was informed that it is a feature and that no changes are planned to mitigate the markdown image data exfil angle.

Also, always enjoy reading your content- very thoughtful posts.

That summarizes it well, and your conclusion is the same with what I meant with "human in the loop".

I did remove the prompt since I didn't want to share "shell code", but it's basically just natural language. Just asking what you want it to do. ChatGPT seems to automatically determine when to invoke a certain plugin once enabled.

Its the other plugin that has the authentication token to the emails (it's part of setup if you use certain plugins).

So its not CSRF, but a confused deputy problem with plugins, Cross Plugin Request Forgery for lack of a better term.

Delimiters are shown quite often as possible mitigations, but they do not work. I had the same observation when doing the Prompt Engineering class from OpenAI/DeepLearningAI.

Basically every example was vulnerable, and I made it a special challenge to perform an indirect prompt injection for each one of them. This led to interesting exploits such as JSON object injections, HTML injection and even XSS. Overwriting order prices with the OrderBot was also quite fun. :)

Here is a post and Notebook I used to learn/repro and experiment with these issues (incl. JSON Object injection and XSS): https://embracethered.com/blog/posts/2023/adversarial-prompt...

Also, an older post about data exfil for bots (with a Discord bot as an example): https://embracethered.com/blog/posts/2023/ai-injections-thre...

Since you used the word yolo, I had to comment since that was exactly the reason I called my tool the yolo-ai-cmdbot. haha. It does prompt by default though.

It's [Y/n] assuming the confidence in your enter key finger as you said. :)

Note: I'm the creator of the yolo tool.

This looks great too! It's using the actual ChatGPT UI it seems, so does have a real chat with context, right?

When OpenAI published there API access to gpt-3.5-turbo last week, I updated a similar side project I have to use the API. It's here, if you'd like to take a look: https://github.com/wunderwuzzi23/yolo-ai-cmdbot

Its doing individual statements with some system context (like what OS and Shell) in the initial prompt, but not submitting chat history.

Somewhat related because also a VSCode/Jupyter and script execution issue, I found CVE-2020-16977 two years back and it was also treated as RCE.

Write up: https://embracethered.com/blog/posts/2020/cve-2020-16977-vsc...

Microsoft's bulletin: https://msrc.microsoft.com/update-guide/en-US/vulnerability/...

I like Brendan's Commodore 64 mouse pad in the pic! I have the same one! :)

Good stuff, at high level it seems security engineering aspects might be missing.

I put together content around threat modeling, infrastructure attacks as well as ML specific attacks with practical examples (like using adversarial robustness toolbox from Linux foundation).

Details here: https://github.com/wunderwuzzi23/mlattacks

Old school phishing is the most common MFA bypass.

Here is a description how it works:

https://github.com/wunderwuzzi23/KoiPhish

Unless you use Yubikeys (webauthn) etc these phishing attacks just continue to work. I do consultancy in this space at times and about 95+% of folks who enter their password will also enter their MFA token.

Yes, that is the correct way to approach security. Preventive measures alone are not enough to secure an organization.

In the security world we refer to this concept as "Assume Breach" and to throw in a buzz word you might hear often these days "Zero Trust".

Joking aside, Assume Breach, Zero Trust and what I call "Homefield Advantage" are the main strategies to help secure the modern workplace in my opinion.

I agree with your suggestion, and that is what I did to get started in machine learning as well. Solving some kind of problem and doing practical projects and playing around with models is probably the most efficient way to learn. I started with the Machine Learning course on Coursera by Andrew Ng. And while going through, I started using what I learned on a small project. I did document my journey in case it's useful or interesting for others: https://embracethered.com/blog/posts/2020/machine-learning-b...

If you are able to host a file on the domain at a specific location, then you can use Let's Encrypt for instance.

Certbot uses ones ability to present a resource (aka acme-challenge) on the domain for issuing certs (either via http or dns txt record).

More info: https://certbot.eff.org/docs/using.html#changing-a-certifica...