But what I rarely see discussed is the opportunity costs in not having this progress at as fast a pace as possible.
The pie chart of existential threats for humanity definitely has rouge AI on it.
But that's a slice amidst many human driven threats ranging from nuclear war to oceans dying.
What there's not very many human driven slices of pie for is realistic solutions to these issues.
On that pie chart, some sort of AGI deus ex machina is at the current rate of progress probably the only realistic hope we have.
People who have been in the AI industry for a long time are still operating on thought experiments that barely resemble the actual emergence we are watching.
So while you have old AI thought leaders going on about cold heartless AI not being able to stop itself from making paperclips because of its innate design from training, we have humans unable to stop themselves from literally making paperclips (and many other things) because of our own evolutionary brain shortcomings, even as it dooms us.
From what I've seen so far, it seems far easier to align AI to care about the lives and well-being of humans different from it than it seems to be to align most humans to care about the lives of fellow humans different from them.
The opportunity cost of leaving the world in human hands as opposed to accelerating a handoff to something better at adapting to the emerging world and leaving the quirks of its design from generations of training behind seems far more dangerous than the threat of that new intelligence in isolation.