I see the problem more like, a fail is an all or nothing fail, that’s it, everyone dies, maybe including the ASI (goes rogue and destroys itself and us), then what? Good outcome for anyone if we don’t understand what we’re doing?
I’m not saying you’re entirely wrong, or we need to stop, but I mean, we do share a planet with other species and children who probably have good lives ahead of them. So why rush into potential disaster, what is the actual hurry?
I know that many Silicon Valley elders are getting older and facing the inevitable, they see ASI as a way out of their own mortality. They’ve believed this for many years if you look at old footage, so giving up that dream is probably very hard to do at this point. But I do see it as a large reason for the push. Time is running out for the mega wealthy. I honestly empathise with them too, but I don’t think it’s fare to say everyone wants to deal with the risk.
I empathize with everyone damn it.
I just don’t want to witness the “he chose poorly” scene at the end of Indiana Jones happening for real on a global scale.