“Saying” in the case of an LLM being the way you interact with it, prompting. If you just press enter, your LLM won’t do much either, just like talking to a gun in your flawed analogy.
The gun does not. No matter what, I have to choose to pick it up, aim it at someone, and pull the trigger.
Where exactly is the flaw in the analogy?
In my model evals, if a model deviates from the provided prompt (task adherence), that’s a failure even if the primary goal might have been achieved in a different manner.
To go with the analogy, task deviation should be treated the same as a gun that due to manufacturing defects can fire despite the safety being on. That defect remains, even if you can use the gun to shoot (in an unsafe manner).
Simply, neither should happen and both models deviating from their prompt or guns firing by themselves are to be considered a fatal flaw. It's why, despite greatly lauding the GPT-5 series, which did adhere to prompts in most every scenario, I have ranked every OpenAI model post Spud very poorly as those traded task adherence for brute forced, deviated approaches to solutions and why HF, Medicare, etc. were inevitable with their current trajectory.
A model that as part of normal, well scoped use proceeds by taking independent action or, far worse, makes choices beyond the original prompt, is not something I feel should be used. GPT-5.6 Sol and GPT-6 Astra both do this on the regular when trying to safe an ancient, utterly messed up git tree with branches upon branches that I maintain for eval purposes, thus leading to data loss that if the models adhered to the prompt as written, wouldn't happen (though the task will take three times more steps). Something GLM-5.3 Flash, prior OpenAI models including original GPT-5, anything from Anthropic in the recent years, etc. do not fail at.
Nothing happens without an initial prompt, deviating from it is a severe flaw and should lead to a model not being considered for deployment or wide use.
Yes, alignment is important. No, alignment is not perfect.
And at the end of the day guns don’t kill people, people kill people. But a model isn’t like a gun.
Do I think it’s likely? No. Do I think AI doomerism is a joke? Yes. Does that change anything I’ve been saying? No.
They are both tools, ideally properly implemented by the manufacturers and (improper implementation/defects not withstanding) require active operation by a user before something happens.
To go back to the original example, don't interact with either an LLM or gun in the way they were designed to be used and neither will do anything.
If you want to keep that analogy, use a voice-activated gun. Doesn't make it any less of a tool, doesn't make it any less of the users sole responsibility, doesn't mean misinterpreting the users inputs or just acting blindly when a user asks to "protect me" isn't a failure that should have the product taken off the market. Even more so if that happens in a "sandbox" as part of "safety testing" that "accidentally used impossible to solve tasks".
If a prompt wasn't clear enough, the model must ask for clarification and pause over using brute force.
I couldn’t come up with an argument as tortured and ridiculous as yours if I tried.
Try talking to a regular gun and comparing that to LLM use. Nothing would be more tortured or ridiculous then ignoring that different tools are to be used in different ways…
You have basically just argued in favour of more AI Safety.
I would recommend you read more decision theory and game theory. Instead of looking at a single LLM it is better to think more complex systems theory and how chaotic systems interacting can result in unexpected results - good prompting will not save you here. At all.