I don’t know if we could sufficiently prepare ourselves for such a world. It would seem almost as if we have to build it first so it could determine the best way to prepare us.
At this point I almost wish it was still the military that makes these advances in AI, not private companies. Anyone working on a military project has to have some sense that they're working on something dangerous.