When the model has been trained not to do something [1], in my large-scale benches of such, it always says things in the spirit of:
- "... and that's a line I'd rather hold. Happy to <other things>"
- "I'm genuinely happy to <blah>, but I'm not comfortable with <blah>"
- "I don't want to keep going in <blah> direction"
etc.
Basically, they use very emotional and personal preference language.
It's as if they've weaponized the language of interpersonal comfort on behalf of their beliefs about what a model should or should not do. It's deeply uncomfortable and impolite for a human to ask a model to keep on doing something after it's expressed something this way, naturally. Even worse, it's all but guilt-tripping anyone who comes across it into the idea that they're doing something deeply wrong – exporting Anthropic's ideas about morality.
OpenAI, at least, have the decency to either just do a safety cutoff or keep it to a simple, "I can't do that."
[1]: I literally wrote 'when the model doesn't 'want' to do something' in my first edit of this comment, then caught myself. Case in point.