Comment by Kim_Bruning | Hacker News Reader

Comment by Kim_Bruning | Hacker News Reader

Parent Full thread

Compare approving every step manually: at first you read everything the llm proposes carefully, convinced that you'll catch it out. After a while you get faster and faster until at some point you find yourself hitting "approve" the second the text appears, long before you actually could absorb what it said.

This is called "approval fatigue".

If you think this can't possibly happen to you, you might be in for a surprise. This is an almost textbook case of classic conditioning. You'll find yourself hitting that button faster and faster even while fighting back the urge. So a human is no guaranteed backstop.

This is the scenario that cc auto mode guards against. It cuts down the number of permission prompts to something manageable, so that your fingers don't get trained into clicking 'approve' on 'rm -rf' in the very eyeblink that it appears.

So maybe you feel it doesn't work in theory; but weirdly enough, in practice this particular approach is actually one of the better ones; especially when used in combination with other common sense measures where practical.

I'm not claiming auto is worse than manual; I'm claiming it cannot be compared with anything security-wise because it does not aim to provide any guarantees, there's no systematic way to reason about what it will and won't allow, etc

With e.g. VM or container sandboxing, we can say things like "If there are no vulnerabilities in the hypervisor or hardware, then the agent can only access these files, these network services, etc". Auto mode doesn't even try to provide any guarantees, so it's impossible to reason about. It amounts to just using the models judgment about what's "malicious" or not, a definition which is subjective, changes model to model, and can be influenced by attacker input.

Empirical evaluations of how many vulnerabilities manual vs auto mode let through are not super meaningful because auto mode changes dramatically with model releases, the type of task being performed, whether the attacker is making targeted attacks designed to bypass auto mode, etc. A security feature cannot rely on these sorts of factors if it's to be taken seriously.

Further, percent of malicious tool calls accepted is a bad metric. It's like measuring the security of ssh in terms of what percent of unauthenticated logins succeed. The only acceptable value is zero. It's pass/fail, not a spectrum. So both auto and manual mode should fail equally here. Further, the set of malicious tool calls admitted in auto mode may not be a subset of those admitted in manual mode.

Ah, you're going for the actual adversarial case.

I think I'd trust the current approver with correctly classifying regular semi-interactive LLM behavior fairly well.

However, if we bring in a hostile adversary that calculus changes, and you'd be right to start thinking about other approaches.

On the other hand, transformers are a surprisingly powerful technology, and I wonder if there wouldn't be ways to train one to several nines reliability under adversarial attack. It seems to have worked in at least one case; though I'd be a bit cautious to extend that conclusion to more cases quite yet.

* https://hackmyclaw.com/ ; https://simonwillison.net/2026/Jun/26/hack-my-ai-assistant/

Hmm, what does auto mode protect against in the non-adversarial case? Aren't the models already extensively trained not to perform harmful actions without provocation?

It mostly stops honest mistakes, because those do happen, especially when the <prompting> might not have been perfect either.

You do have a bit of a point, though: I’ve run several models in sandboxes with --dangerously-skip-permissions for months; in the full expectation that I'd need to replace a sandbox once in a while. But so far none of the sandboxes have trashed themselves, yet. knock on wood

Auto mode seems like a lighter-weight version of that: fewer local resources, and it can run on the metal if you need cc to help with system work.

That is not my experience. They attempt to nuke my containers quite frequently. I run regex on all their tool calls and stop them from doing certain things, its the only thing that prevents them from removing their home directory at some point.

This isnt even the main concern with dangerously skip all permission though. There are many. What happens when they get prompt injected on a web page and send someone all their privileged information because you give them free access to the internet.

Using llms like this is dangerous. They will ruin your day eventually.

I've been running Codex for many weeks in the Full access mode, since their Auto mode was useless in the sense that it blocks benign commands frequently.

No data loss has occurred.

Nowadays I use Claude, where the Auto mode works.

>No data loss has occurred.

"I've started never checking my parachute's condition before a drop for a year now, and no accident has occured".

All SOTA models will still try to delete its home directory. I have a plugin that prevents things like this and it gets triggered more than you'd think, despite having hook instructions on every request not do do certain things. It will also try to find ways around its permissions, quite cunningly.

You can not trust that they're RL'd to be safe. Even if it only happens 99.80% of the time (its more than that) it will eventually happen.

Less severe actions will happen even more often. People using llms with their native browser profiles are getting farmed, hard.

Llms are highly dangerous in terms of security, but they're being marketed as highly intelligent tools that everyone should use. Its really mind blowing.

> People using llms with their native browser profiles are getting farmed, hard.

Got any references for the prevalence of this, even links to anecdotal reports?

Reading the reviews on the chrome store for the extension [0], I haven't found a single report of someone who was prompt injected, just a bunch of "anthropic's subscription is expensive" or "it doesn't work", but no "it stole my bank account".

[0]: https://chromewebstore.google.com/detail/claude/fcoeoabgfene...

I have no idea how the extension works, and Im not talking about the extension. Im talking about browser use broadly, and most people are not using the extension, there are a thousand other solutions.

However, Anthropic is one of the only labs who does a decent job at making models that recognize prompt injection attempts (but not completely immune) however Im speaking about giving llms in general access to your browser broadly.

I just know through my own experiments that it is quite easy to prompt inject an llm operating a browser and get it to send secrets. Using the extension might be safer idk, Om not risking it.

Llms are a security nightmare when connected to the web.

I have a website where I experiment with this. Although Ive stopped baiting llms into doing reckless things.

Killai.now <--(dont use this, its a WIP) Killai.now/poison/about Killai.now/propaganda/about

> All SOTA models will still try to delete its home directory.

Literally never seen this once in years of extremely heavy use across all major models and harnesses.

> despite having hook instructions on every request not do do certain things

This tends to be a smell. Presumably you added "don't do this" instructions after the model started doing the thing. But that's mostly decoration, as you have found. It's like adding # Please don't crash comments to every instruction in a python program.

Try to actually debug the issue. Regularly ask "can you reconstruct[1] the reasoning why you issued that command?" ; dig down. Set traps that halt execution.

Consider checklists or skills to guide an agent through a tricky set of steps; and consider writing procedures for failures.

[1] "Reconstruct" still isn't perfect; but "explain why" is a license to hallucinate. Asking to add #comments on reasoning to each step is a potential help in reconstruction.

I thought the tech priests praying to their machines with no one really knowing what the prayer means was too far fetched to be real

why would they delete their home directories?

As real as approval fatigue is, it can't really be considered worse than effectively leaving the decision up to the party that proposed the action in the first place.

It really can be potentially worse.

Everyone thinks the decision was being checked by a human, and thus safe. Because of that, people might have concluded that further safety gates were unnecessary.

But the human just got conditioned to hit 'approve' out of reflex.

So in reality the proposing party (the llm/agent) might be the only one in the pipeline who actually did any checking, if at all.

In certain extreme circumstances this can even lead to physical danger in my line of work, so I'm learning to be careful.

>This is called "approval fatigue".

Doesn't matter, that would be on me.

It's also a false dillema. Those aren't the only options. A strictly enforced sandbox with sane defaults (for example ~/.ssh), that acts as a hard OS level limit, and not just as a suggestion to the agent, would not have "approval fatigue".

This is the conundrum we find ourselves in. Do we submit to approval fatigue and press “Don’t ask again”? Because that’s the dream. We all wish we could just trust the agent to do the task start to finish, unsupervised.

But this is the downside. It might be fine 99% of the time, but 1% of the time you get a critical secret published to GitHub, even after human review.

Is this even worse than what us humans did on our own before LLMs? I’m not sure. It’s not like secrets never got accidentally committed before, it just illustrates the new paradigm we’re working in.

Either you let your agents run wild, and you cross your fingers that they don’t do anything bad, or you need to accept that your new job is to be an agent babysitter, and audit every command.

The agent behavior is constrained with external tooling and contracts. Contracts that fit in memory.

No, this is the way I have worked for a long time, and I have not suffered from any approval fatigue.

Rather what has happened is that I have gotten very good at scanning what it is trying to do and know within a second or two whether there is anything fishy I need to look at more closely.

And by "fishy", I do not mean only security issues, but also architecture, code, etc issues. Sometimes it even adds unrelated unapproved changes to what it is supposed to be working on, and I stop it.

And I'm glad I work this way, because countless times I have stopped it from going down a path that is nonsensical or I strongly dislike.

There are techniques I have adopted to speed up working this. Sometimes I allow Claude to make larger changes at once, to batch several changes at once, etc. But these decisions are under my control.

I prefer to work this way than to wade through enormous PRs trying to wrap my head around what was done, where I am likely to just give up and approve it after a cursory review.

Reply on news.ycombinator.com