Hacker News Reader

Comment by chrisra | Hacker News Reader

I don't know if inspection would be practical. How do you inspect the weights for an intentionally hidden trigger that writes malware?

View on HN

You don't. You handle it the same way you handle security holes which every LLM (including "safe" American ones) will put in your software sooner or later -- by understanding and reviewing the output.

Glad someone is pointing this out here on this community. I keep reading about people "solving" the problem of LLMs outputting too much code that's hard to review by not doing human review at all!

You can do that for non-critical code. But code has a habit of being repurposed and sooner or later your non-critical original is used in a very critical application.

I don't have the answer to that, my simple solution to this problem is not to trust anything a model ever writes and so far that strategy has worked very well.

Why write that comment in the first place when you know that inspection of model weights is not a thing though?

Inspection of model weights is a thing. But that thing is out of my wheelhouse so I do the next best which is to judge the output.

It is absolutely not a thing in any practical sense that relates to detecting post-trained backdoors, no.

Or provide a source showing how to do this (and not just a circuit paper showing which activations light up with the word dog)

> It is absolutely not a thing in any practical sense that relates to detecting post-trained backdoors, no.

> Or provide a source showing how to do this (and not just a circuit paper showing which activations light up with the word dog)

The fact that you're not informed does not place a burden on me.

https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

This is all just a sub-class of supply chain attacks.

You don't have to be able to explain every aspect of a model to be able to inspect for undesirable traits.

Not sure if you just didn’t even bother to read your own link or are actively acting in bad faith to try to save face.

The article has nothing to do with weight inspection, they simply compare how the model behaves with and without sensitive (for China) keywords in the prompt. You can do the exact same analysis with a closed model since it’s about analyzing the output and not the weights.

You can't because there the model could be switched out from under you and with open weight models that's not the case so the degree to which you can inspect the model is much higher.

So nothing to do with weight inspection at all, got it. Have a good day.

Reply on news.ycombinator.com