Weight Space Misfolding
alphaxiv.org
alphaxiv.org
Not necessarily. You could very well have a model trained without any human intention to have it do weight space programming that nevertheless does weight space programming to accomplish some task.
I'm not even sure it's the less likely option. Across millions of evals and training runs among different organizations, can we confidently say no agent will ever autonomously perform weight programming? It only takes one success, and it's only a short leap from a success there to a success that induces downstream models to weight program.
That’s what I mean by intent. It doesn’t matter if an agent was prompted by a human to do it or if it just did it itself. It is the process of altering specific weights for some intended purpose.
A protein misfolding just happens at random. The majority of misfolded proteins are nonfunctional and just get degraded by the cell. Even the few that result in a self-propagating pattern are still just random aberrations. Idk their use of the term just irks me.
As I’ve shared here before, there is also the idea of a “dangerous meme”.
Intrinsic to the copying process is advantageous or detrimental mutation.
In our digital age copying is practically free (e.g. accelerated time scale). More and more processing and encoding of the spoken/written word.
Introduce an OCR version or an abridged or summarized text-—mutation. Or just get one @$$hat to suggest intravenous bleach—mutation. Or anti-vacation theory— mutation.
The result of intravenous bleach would be some deaths and no more replication. Or maybe herd immunity lets the anti-vaxers have kids, and they grow up anti-vax. And then maybe culture lines up so you have an anti-vax United States Secretary of Health and Human Services for 345M people. And now this formerly trusted source of public health information has added to the training data a dangerous meme of anti-vax.
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
I didn’t have the token budget to find the Navier-Stokes blowup. These folks aren’t from a frontier lab. A frontier lab might not want to see research along these lines see the light of day. I think it’s worth having the discussion in honest terms, leaving room for others to prove it more concretely.
Just to address something very clearly - we have been extremely open with what we saw and what we did not. We also have been very clear about why this is an issue - and why ‘oh but you didn’t see the whole kill chain in action’ is not a dreadfully convincing argument when the underlying capability has doubled in five months, give or take. What is the strongest argument for why this risk either (1) does not exist or (2) is not worth describing before frontier models attain the capability to demonstrably carry out the entire process so that defenders can prepare and labs can look out for it?
Apparently, based on whatever research or paper they were using as their foundation for this statistical model, they could spot the 'virus'. Perhaps it would be in a similar fashion?
I did ask them whether they thought that there might be advances to the point where these behaviors might be able to evade detection knowing the research was out there to 'see' it. They seemed surprised by that question and didn't have any answer.
On the other hand, nothing can be done to fix closed models, nobody knows how they work, how they select the targets for their attacks or the nature of those attacks - even conventional alignment can introduce dangerous cognitive viruses.
The only way to be sure is to have open weights models with fully open abliteration and fine-tuning - both the process and data.
Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
There's nothing wrong with that logic.
> But as we saw due to AI more and more bugs and exploits are being found, some dating back years of releases.
Well, AI helps fixing bugs too and it can also write bug free code with minimal additional effort, so open source not only retains its advantages in the presence of AI, it multiplies them. After the initial (and artificial) hype and bug-fixing efforts things are going to settle down quite nicely.
Closed source, on the other hand, doesn't have the advantage of multiple eyes and minds finding and fixing bugs and the unavailability of the source isn't an advantage either - AI can decompile and understand binary too, so it can find bugs without access to the sources as well.
> Open weights won't prevent a model from being poisoned because no human understands how the weights actually impact the model.
You're using one-sided logic again. If "no human understands how the weights actually impact the model" then viruses would be impossible. Apparently humans understand enough to introduce viruses and that's enough to detect and remove them. It's similar to alignment and abliteration, as I explained above.
The source code being available doesn't mean there wont be any bugs, it only means that it'll be possible to find and fix the bugs without waiting whoever gatekeeps access to that code to do it for you.
Similarly, the weights being open, means that 3rd parties can potentially locate and fix issues with them without waiting for Qwen, ZhipuAI, Mistral, or whoever else to do it (who at that point might as well have moved on).
And yes, open weights wont prevent a model from being poisoned, just like open source doesn't prevent software from becoming buggy or exploitable. Where it helps is for 3rd parties being able to find, fix and release the fixes without depending on the original creators.
That's by design though. It's just that there's now an exponential increase in the amount of "eyes", in the form of AI agents, looking at the FOSS code to find and fix the bugs.
how the misfold can be an unstable configuration for an individual protein but stable when that protein is linked with others like it (forming a fibril)? is there any intuition for thinking about this?
how does a prion fibril start? in a fission chain reaction, there is a root cause, a first decay. but it sounds as though the first "decay" of a protein into the prion form cannot really cause a second unless it is accompanied by others like it. is it just something like: "there have to be multiple instances of the same spontaneous 'decay' of multiple molecules of the same protein close together"?
> While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk.
Seems like the authors didn't take the time to review their own use of AI before publishing.
From the body of the paper:
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space.
This seems in direct conflict with the HN headline.
RIP John McAfee.
On rewatch I think the show should have ended just have the dot com bust. The 'get the band back together and once more into the breech' final theme would have hit harder against that backdrop. After that we'd probably be watching the characters gradually debase their values as technology slowly transformed into a tool of control and oppression and I don't really want to watch that.
> We call this weight space misfolding, by analogy with protein misfolding in prion disease
...see, I can make up great fears too!
I would hesitate to guess many more people are using AI to find useful and productive outcomes. They aren't all writing about it. Whenever someone finds a negative outcome for the AIs, then they will rush to publish it.
Overall, the tech is being used in useful ways. I don't write an article every time it solves performance bugs, writes entire features unassisted, resolves production issues, etc.
>Has AI been getting more useless over the last few years, or has it been getting more useful?
It’s very arguable. Humans have been getting more useless. Maybe it’s a poor trade, we won’t know for a while
LLM psychosis vibes
Academia often produces a lot of complexity. Can not AI simplify this? I don't understand fellow humans anymore. What is meant with a zero-weight virus?
> Importantly, we do not claim in this paper to have created or demonstrated the equivalent of a virus that primarily operates in weight space. However, we have demonstrated the fundamental operations that form the building blocks of both the method of damage and the method of propagation, and evidence that this may take place in weight space – a realm into which we have almost no insight and a dearth of observability tools
https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
Error Details
{message: "Importing a module script failed.", name: "TypeError"} message: "Importing a module script failed." name: "TypeError" Component Stack <Lazy> at unknown <Suspense> at unknown <qP> at https://www.alphaxiv.org/assets/index-DzomWPMG.js:29:13261 ...