Comment by arthurcolle | Hacker News Reader

Comment by arthurcolle | Hacker News Reader

Parent Full thread

I baked in lethal trifecta guards into my primary harness, DSCO, but it makes things much less smooth to run autonomously. Basically if you load secrets into a context, ideally it stops that and creates a process running a small local model that will write some code to not look at the secret and instead run code to operate on the secret without reading it in.

The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets

You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.

Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.

Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here

Writing on mobile so this might poorly articulate the issue but it's interesting

What stops the agent from just writing code to actually read the secret? That’s kind of the core of the problem. The agent can be tricked into doing anything.

Indeed. But if it's a model running on your own hardware it isn't as risky. And the Anthropic models are actually pretty good at kind of writing code that will look at contents of a file without fully looking at the content, so it's insecure but not quite as bad as an open("secrets.txt").read()

What does that mean? That it only reads part of it into context or RAM?

Local model or not, all LLMs are susceptible to prompt injection and making mistakes (like those cases where you see the agent randomly deleted someone's home dir.) That's what Goldman are likely worried about. Not Anthropic going rogue or something.

Concealing API keys is a relatively minor protection in the wider picture of the risks.

What does using a local model have to do with it? I’m not worried about the secrets going to Bob, I’m worried about them going to Eve.

I wil encourage you to check https://runeward.io this is specifically the usecase we are building this for

I don't know about GP's setup, but you give the AI an MCP that can do the thing without exposing the key to the AI directly. An all powerful agent could setup a fake endpoint and send itself requests and exfiltrate the key that way, so don't let it do that.

Uh why aren’t you using allowlists to prevent exfiltration to random domains?

I do this at the tool registry layer but this is a good idea for the agent sandbox itself

If you give access to bash, cc or wget then you're wide open, no matter how much you tell yourself you're safe.

Reply on news.ycombinator.com