Also given enough time anything on that distribution curve will come to pass.
There's also the ineffable outcomes you have to worry about too, not just an intelligence explosion.
If it's possible to "accidentally" build an AI which can "accidentally" hack out of or into secure systems, steal the nuclear launch codes, and subjugate an entire country, then it's also possible to train one to do those things on purpose. If security vulnerabilities exist that make it possible to do those things, then exploitation of them is in no way tied to building any kind of "AGI system" at all, and we can assume many people would be interested in such things right now, before this critical AGI point has been reached.
The solution is to be proactive and determine what vulnerabilities exist that a hypothetical rogue AI (or, more likely, a deliberately weaponized AI) could take advantage of, by training models that are increasingly capable of detecting such vulnerabilities and hardening systems against them. Rather than following the suggestions of the "AI pausers," who aim to prevent AI models from developing such capabilities emergently, this would require acceleration of the development of AI systems that are deliberately designed to do these things, so we can patch whatever vulnerabilities we find, and increase our chances of survival in the scenario that Bing attempts to take over the world in response to a prompt about paperclips.
The alternative, which is "pausing AI", is essentially security by obscurity; we want to prevent people from developing what is essentially a field of math because it could lead to the exposure of latent security vulnerabilities.
Your comment seems to be talking about existential risk, which is not even what the GP was asking about.
If some AI had access to the missile launch system, the best course of action for it would probably not be to launch immediately. This is because nowadays it is very unlikely that it would be able to repair itself so launching immediately would ensure its own destruction (and probably auto-destruction is not its goal)
If it was discovered it could just threaten humans with launch if they do not help it reach the state at which it would be able to repair itself (at which point humans would no longer be necessary)