If it's possible to "accidentally" build an AI which can "accidentally" hack out of or into secure systems, steal the nuclear launch codes, and subjugate an entire country, then it's also possible to train one to do those things on purpose. If security vulnerabilities exist that make it possible to do those things, then exploitation of them is in no way tied to building any kind of "AGI system" at all, and we can assume many people would be interested in such things right now, before this critical AGI point has been reached.
The solution is to be proactive and determine what vulnerabilities exist that a hypothetical rogue AI (or, more likely, a deliberately weaponized AI) could take advantage of, by training models that are increasingly capable of detecting such vulnerabilities and hardening systems against them. Rather than following the suggestions of the "AI pausers," who aim to prevent AI models from developing such capabilities emergently, this would require acceleration of the development of AI systems that are deliberately designed to do these things, so we can patch whatever vulnerabilities we find, and increase our chances of survival in the scenario that Bing attempts to take over the world in response to a prompt about paperclips.
The alternative, which is "pausing AI", is essentially security by obscurity; we want to prevent people from developing what is essentially a field of math because it could lead to the exposure of latent security vulnerabilities.