Hacker News Reader

Comment by hypfer | Hacker News Reader

True, and cool that someone actually put in the work to prove it, but the takeaway is still kinda wrong?

Because it's not the abliteration that does this. It's the "untrusted artifact" that does.

Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).

That seems like a pretty obvious thing to mention, as it is a clear consequence of the research they did.

___

Add to that, that it's not actually the "artifact", but the "in-memory representation", meaning that anyone or anything that control that can _also_ do this. Be it your inference provider, or, a level further down, the firmware of the GPU it runs on.

View on HN

Yes. You get the model from HF? HF could've edited the weights.

You use an unsloth quant? Unsloth could've edited the weights.

You create a quant yourself? Your version of llama.cpp could've edited the weights.

And then yeah, as you say, the actual inference engine itself, the firmware, even the hardware.

Untrusted is untrusted, abliterated AI or otherwise.

it's the same "trusted compiler" argument, only at a different level. Though at least here you could -- if you wanted -- check the generated source.

> Because it's not the abliteration that does this. It's the "untrusted artifact" that does.

> Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).

Does it even need to be untrusted? From the article apparently it only takes ~250 poisoned documents in the training data to backdoor a model, and it survives safety training and fine tuning.

This has been known for over a year: https://www.anthropic.com/research/small-samples-poison

Which means any model trained after 2026 is succeptible to slurping up poisoned documents on the internet. We all know how prolific the crawlers are in a desperate attempt to slurp up everything on the internet. And all it takes is the lab not catching ~250 documents.

So these agents that can process anything on the internet through an LLM trained on the internet only have to stumble across the payload. Seems like agents running through LLMs is a terrible idea from a security perspective.

The idea that any LLM trained on public data in 2026 or later could be invisibly poisoned in multiple ways by multiple parties is horrifying.

If it actually works that way, I think it's almost guaranteed organizations and people will be doing it, and soon.

totally agreed, and..

>, the firmware of the GPU it runs on.

that'd be a rad PoC to see, but given the gpu values right now i'll let someone else brick things.

"Even a "clean" model of a lab could be doing this."

The risk is in using a language model instead of using a brain

NB. This comment expresses no opinion on whether the risk is or should be "acceptable" to any particular person or not

This is why I absolutely will die on the hill that models can't be called open source if we don't have the source that created them.

I understand this in principle, but I've been wondering: what guarantee do you have that the sources that are claimed to be the training are the resulting model weights, without interference?

Maybe you trust the creator more, when they release the training materials as open source, but I don't see how you'd get a guarantee.

md5 hash checksum

Checksum of what?

Models are not reproducible.

Sure they are that's what weights artifacts are surely?

Say I'm a lab.

I train a new model. I publish my training data and my training code/workflow.

I then secretly do additional post-training to poison it. I do not release the data or workflow for this.

Now, I publish the poisoned weights and an md5 hash of the poisoned weights.

To you, the user - you have no way of knowing this happened.

You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.

If the original model is poisoned then yes a checksum etc doesn’t matter. If you as the end user are being diligent then you are downloading the checksum and checking the download against it for precisely the scenario you talk about.

Personally I find it a bit exotic as I’d probably just stop trusting the vendor first but checking against a checksum does stop attacks where others alter an artifact.

If making sure the download is intact is what the above user wanted a checksum for, they're solving entirely the wrong problem. And your followups here are also solving the wrong problem.

The problem we want to solve is making sure the training data matches the model. The only use of a checksum that helps that problem is if multiple entities do the same training and then make a checksum of the result. But that requires training to be completely reproducible, which it's currently not.

They absolutely can be made to be.

How many billions of dollars would that cost you?

In a big LLM we are talking 10^24 operations or more. When you start talking about error rates per trillion flops, and talking about all the potential ways to compute operations in different orders you are talking about one of the more difficult things undertaken by humans.

A checksum on the weights should suffice same as any other artifact.

Exactly this.

The same is true with open source software that distributes binaries: you can recompile (retrain) it yourself, but if you choose the convenience of the binary (final artifact) you can't guarantee one comes from the other.

It's even worse because AI might be able to tell you that the source matches the binary, but it can't do the same with a LLM.

Just spitballing-- you'd see other firms generate their own versions with slightly modified training sets. If you were a large org that wanted to do that, you might want to also reproduce the original model in order to chase down any quirks you see in your model. Then you'd see the weights don't match up.

Or adversarially-- if e.g. "Deepseek Open" became one of the most widely used commercial models, it might behoove the US government to reproduce it as a check against intrusion, and vice-versa for China and any American model.

I guess you could do some SNARK like proof. It would add a bunch of overhead and probably influence how you architecture the model though.

rather than that, what guarantee do you have that the model does what you want under the conditions you care about?

how do you know you are outside of those conditions?

im not convinced that having the training data and process actually answers the most important questions, because we dont have answers to them.

the rest is great for being able to audit what is there, and being able to play with different variations, but it isnt going to be the same as reading code, then building it, then comparing the build hashes to see that they're the same

Reply on news.ycombinator.com