Imagine a mob boss whose spine was severed in an unfortunate mob-related accident. The mob boss cannot move his arms or legs, and can only communicate through speech. Said mob boss has it out for you. How worried are you? After all, this mob boss cannot do things. They create speech in response to prompts. And that's it!
I actually don't agree with Eliezer that the primary threat model is a single consequentialist agent recursively bootstrapping its way to uncontested godhood. But there is a related threat model, that of "better technology allows you to make bigger mistakes faster and more vigorously, and in the case of sufficiently powerful AGI, autonomously".
In terms of proof that it's possible to destroy the world and kill all humans, I will not provide that. No matter how poetic of an ending it would be for humanity if it ended because someone was wrong on the internet, and someone else felt the need to prove that they were wrong.
And your mob boss example is a good reason why: it doesn’t extrapolate that much. There is no case where a mob boss, or a disabled Hitler for that matter, can kill everyone and ends humanity.
The extent to which it's possible for a very intelligent AI with limited starting resources to build up a supply chain which generates GPUs and enough power to run them, and disempower anyone who might stop it from doing so (not necessarily in that order), is a matter of some debate. The term to search for is "sharp left turn".
I am, again, pretty sure that's not the scenario we're going to see. Like at least 90% sure. It's still fewer 9s than I'd like (though I am not with Eliezer in the "a full nuclear exchange is preferable" camp).
But here is the thing: AlphaGo and subsequent AI didn’t make the previous human knowledge wrong at all, most of what was figured out and taught are still correct. There are changes at the margin, but arguably the human are on track to discovered it anyway. There are corner sequences that are truly unusual, but the big picture of playing style and game idea are already on track to be similar.
And it matters because things like nanotech is hard. Building stuffs at scale is hard. Building factories at scale is hard. And just because there is a super intelligence being doesn’t mean they become a genie. Just imagine how much trouble we have with distributed computing, how would a cluster of computing gives rise to a singularity of an AI? And if the computer device has to be the human brain size, there is a high chance it hits the same limits as our brain.
To the extent I expect doom, I expect it'll look more like this[2].
[1] http://incompleteideas.net/IncIdeas/BitterLesson.html
[2] https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-...
Humans are very good at producing CVEs, and we're literally training models to be good at finding exploits: https://www.microsoft.com/en-us/security/business/ai-machine...