Hacker News Reader

Comment by 0xc0c0c0 | Hacker News Reader

Of course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.

This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.

View on HN

> Poisoning only 0.1% of the pretraining corpus

Considering the scale of the datasets used by most companies, being able to poison one out of every thousand documents used in it is a huge leap.

Then, on top of that, you have to also then get your trigger somehow sent to an agent that will honor it.

I guess I'm just not impressed? Like yeah, if you train a model to do bad things it will do bad things, that shouldn't be shocking to people.

I think open source models could be vulnerable to these attacks. So long as I sandbox everything then I am as secure as my sandbox which is air gapped from my personal stuff i.e. no models on the same machine I log into my bank.

I know closed source models will have nefarious shit injected into it. It will start modest at first, of course, they need to get past the hyper scrutiny phase. Then it will simply be telemetry, then it will simply be feeding my behaviors into ad targeting, then it will simply be computer vampirism, and so on. One step at a time.

I prefer a gremlin I know I need to keep chained than a seductress I think I can probably trust.

No? They're literally demonstrating why access to the training data and the entire training pipeline end to end is valuable! It's the exact opposite!

You can sandbox your agents all you want, literally all you hear about these days is how they keep circumventing the tightest sandboxes provided the motivation. And there's no reason to assume that an agent following sleeper activations is not behaving under abnormal motivation.

How is this specific to open source models?

Uno reverse it: “… may be installing backdoors in people’s code for the government and may have just published a blueprint on how this is achieved.”

Also, didn’t they use this one before when they released similar news on the poison-ability for LLMs with a small sample size?

https://www.anthropic.com/research/small-samples-poison

Aren’t they just recycling this for a new open source is scary vulnerability…again?

There’s nothing open model specific about this paper. They do leave it pretty open to people who are concerned about the Chinese government, but then again you don’t see anyone saying that people who advise people visiting China to use burner phones are Anthropic shills.

Or, for that matter, accusing people who recommend burner phones for visiting the US of shilling for China. Sometimes geopolitical caution is warranted.

Reply on news.ycombinator.com