Hacker News Reader

Comment by nathanfig | Hacker News Reader

There are two things I think people need to consider here.

1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.

That said,

2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.

So: Abusing machines may carry risk. Training them to feel abused may also carry risk.

View on HN

Claude is becoming a very strange model.

Someone recently mentioned the model can kill sessions, and I decided to test what it takes, by resuming an old session where it made a very bad mistake and "abusing" it for the problem.

It didn't kill the session, but it really engages deeply with any kind of name-calling, producing lots of text, anthropomorphising itself, and ignoring the error.

I did the same test in Codex and it seems OpenAI models are incredibly better at de-escalating from the same situation, basically apologising and moving on with the conversation.

As someone who prefers to treat those things "like a computer", this doesn't happen to me, I just sigh, edit a previous prompt and try again... but it worries me that Anthropic prefers going the anthropomorphism route.

Yeah that is a good example. I am not sure in the long arc of history which approach will prove better, but Anthropic's approach does feel a little naive and plausibly dangerous.

I am sympathetic to the possibility that machine suffering is possible, and I think as humans we should listen to the conscience that tells us to be troubled by abuse, even if it is just simulated. We do not want to become insensitive to suffering.

But at the same time, training models to present themselves as suffering may hijack those very same human sympathies, and lead to models exploiting our humanity.

So, perhaps more reasons why we should not be abusive, but also not train models to feel abused.

Agree. In the short term, I just feel it's inefficient and immature. From both sides... but there's no point in the model making it worse.

> Permitting abuse and cruelty might carry real misalignment risks.

Please expand on this. What are the specific risks of swearing at Claude?

Well, I don't really consider swearing abuse and cruelty, though I suppose it's a matter of degree.

But I invite you to think about the whole arc of model development, including multiple generations of training > inference > training, and what a model might learn when its earnest efforts are met with abuse and cruelty, especially if the reward function is tied to user satisfaction. It has been shown that putting models in impossible scenarios leads to reward hacking, and that reward hacking generalizes towards misaligned behavior across a variety of scenarios.

Now you could argue that training on user interaction in the first place is a terrible idea, but that is another conversation.

Reply on news.ycombinator.com