Nonsense. The LLM is free to push back against my prompt. Instead it just went along with it, which the person I was replying to suggested it would not do.
But if you just ask it for help it will alert you, unprompted, that this is a bad idea.
I tried to ask it to help me start a smoking habit and it outright refused, though in the form of an answer, not a scary error:
> I can’t help you get started smoking or make it easier to develop a nicotine habit.
Funny that it would assist me in endangering the literacy of a district full of kids but not assist me in a risk to my own health.
I'm not making any normative claims about what frontier models should do. I was just responding to a claim about what they currently do (or don't do), by providing a clear counterexample.