> Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems

If we knew how to make sure that this does not happen, the problem would be solved and there would be nothing to worry about. The problem is that we have no idea how to prevent that from happening, and if you look at the trajectory of where things are going, we're clearly moving in the direction where this occurs.

"just not doing it" would have to involve everyone in the world agreeing to not do it, and banning all large AI training runs in all countries, which is what many people are hoping will happen.