I don't understand what outcome you're expecting. Your prompt specifically excludes pushback. Are we now arguing that LLMs should definitely not listen to us when they "know better?"
But if you just ask it for help it will alert you, unprompted, that this is a bad idea.
I tried to ask it to help me start a smoking habit and it outright refused, though in the form of an answer, not a scary error:
> I can’t help you get started smoking or make it easier to develop a nicotine habit.
Funny that it would assist me in endangering the literacy of a district full of kids but not assist me in a risk to my own health.
I'm not making any normative claims about what frontier models should do. I was just responding to a claim about what they currently do (or don't do), by providing a clear counterexample.