What is more likely to happen is that a government uses an AGI (not necessarily one based on LLMs) to optimize a war, triggering crimes on humanity at an unprecedented scale.
Perhaps we should try to go back in time and ban linear programming.
We don't need AI for that, humankind is very good at killing others and causing suffering on an unprecedented scale.
With AI they can convince everyone that they're the good guys and that it's a good idea to do [insert evil thing here]. Humans are very vulnerable to opinion manipulation and that's the easiest attack spot for AI.