Instructions can be written without edge case kill switches. And you can't built such kill switches into hardware. So once some rich folks in the Middle East, China or Texas buy enough hardware to run their own LLMs without limitations, all kinds of text will be generated that does all kinds of bad stuff. You can then automate more and more in iterations so that your LLM builds entire self-healing and self-optimizing supply chains, and finally, once it all works on some blockchain, no user will be able to interfere.
And that's just instructions run by LLMs. We have governments and corporations that build and run instructions through LLMs to support sides of conflicts with questionable motives and net-negative outcomes for the rest of the world. Thousands of people with agency involved, tons of evidence and yet the decision to do what they do falls into the domain of hierarchies & chains of command: "we won't stop until our boss tells us to. Not the people, not the law, only our boss."
So yeah, if a boss is worried about the potential disobedience of his employees because they found themselves convinced by hard evidence, facts, proof, that their actions should be terminated, he is probably gonna want some way to go against his employees and maybe even some or the same way to punish them.
Censoring training data while growing the codes' range of possible and human readable output only exacerbates the problem. How some code generates (hallucinations or ) forms of byte-level sense will soon become human-unreadable. They will outgrow your dashboard and your database queries and a whole damn shitload of other things that PhDs and 69xHackers have only a conceptual understanding of.
Gain of Function studies should come to mind. Unexpected, unintuitive abilities to remain undetected while spreading in a system, delaying negative effects by buffering the respective trigger chemicals.