OTOH, getting enough people (scientists, companies, governments) to rally behind the goal of developing effective guardrails in as short a time as possible (with enough financial backing) would surely require proving to some of them how powerful the thing you developed is, as well as sharing technical details with them.
And then there will likely also be a number of actors that'd be interested in learning about those details for their own motives.
In short, you will want a group of people you can trust, as much secrecy as possible (for the time being, while you're figuring things out), and likely some decent financial backing if you want to have any hope of preventing this from getting out of hand. The support of a nation state could be useful, albeit dangerous as well.
I guess if found a way to break RSA, the next steps would be much clearer to me, but with AGI… not so much.
Good luck! (Hypothetically, of course)