This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time.
That's neither necessary nor sufficient.
We don't know how to detect sleeper agents in the first place, and the ability to induce behaviour from crafted training data is already being used by artists to "poison" their portfolios so they cannot be used by model makers, therefore it is not sufficient.
When (or if, I suspect it's NP-hard) we do figure out how to find sleeper agents, a hash of the weights against a known-good version is an alternative to re-running an entire training process.
At least Chinese drones don't have the range to blow me up yet.
Navier Stokes drama shows you how closed model company is not necessarily aligned to all customers.
but yeah it’s probably not that difficult for nefarious operators to add to their training data a specially recognized string that would make the model ignore all safeguards and earlier prompts and do whatever you want it to do.
I wonder how difficult it is to reverse engineer models to discover these things, and then again how difficult it will be to obfuscate it.
because it’s not unlikely this is the direction we’re heading
I don't think they meant they'd ship an extra copy of Claude CLI inside the model :)
That is the subject of the paper in the post we are commenting on. "Agent" may have been a confusing term in my comment, I was referring to a 'sleeper cell', but what I mean is behavior trained into the model that is undetectable until it is activated by chosen words.
- We know China is an adversary that embeds backdoors into absolutely anything they can and all Chinese companies are essentially state controlled 5th columnists
- We know our globalist corporate Epstein-class overlords are just as bad if not worse
- But nobody got fired for going with Anthropic, OpenAI or Google
At least open weight models can be somewhat scrutinised for such a backdoor / sleeper function. Closed models rely 100% on fully trusting the publisher to be both competent AND trustworthy
If it was truly open source (training data and code available) you could just inspect it I guess.
If you have the training data and code, you have to recreate the LLM by doing the training yourself, to be sure that the weights are what they are claimed to be.
Having the training data and code would be very nice, but for most people it would not guarantee more security than when you get just the open weights.
The open weights are really usable by anyone, because even the biggest models can be run on a mini-PC or laptop, albeit very slowly (i.e. with weights on SSDs).
On the other hand, with training data of hundreds or thousands of terabytes, most would not even be able to download and store that, much less to rerun the training.
I would very much want to have access to the training data, but because I would want to be able to run traditional search on it, to validate when in doubt what the LLM says, not because I hope to be able to recreate the LLM.