Hacker News Reader

Comment by benlivengood | Hacker News Reader

What I envision is the ability to find the circuits and neurons evaluating morality and ethics and being able to figure out how they work at the boundaries and ensure safe deferral to human judgement or stopping outside of in-distribution behavior, which further is vetted against actual human values.

View on HN

I don't see a lot of hope for that, because it's not a well-specifiable quantity. I would say we currently have no real evidence that you can even do this even for concepts that are much more well-specified like, say, dogs. Maybe you can! But that is not affirmatively known, and nothing we have is on track to supporting that, right now.

The article is literally full of "dogs" neurons and even more specific "boston terriers" and "terriers" neurons in the larger models.

Specifically, there were "lost property" and "honesty" neurons that drove the answer to return a lost wallet in Kimi K3.

That's the big news of this article, that concepts are finally becoming human-legible for many neurons in mechanistic interpretability.

I know, that's why I picked dogs as the example. What are you specifically disagreeing about?

I think the paper says that while there do exist dog neurons that carve out the difference between a "Boston terrier" and a "terrier", those neurons actually only get to carve that fine line because they're part of a whole flow control system where activation has already flown through much coarser grained discriminator neurons (like a "V" neuron), including sometimes through the negative side of activations, before the terrier discriminator ever played a role.

So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.

You’re only considering the front half of the network though, classification is a helpful example for explaining the concept - but if it follows through to the output then you could quite easily imagine just shunting some activation from one branch of the representation tree to another to steer behavior. Which we already do with steering vectors - but if you could read off the actual topology you could do that with much higher fidelity.

Whose values? Yours? Why?

The hard part of AI alignment has never been making models follow instructions. It's that not everyone agrees on which and whose instructions AIs should follow.

That sounds wishful rather than factual.

Reply on news.ycombinator.com