The hypothesized superintelligent AI will be essentially immortal. If it destroys us, it will be entirely alone in the known Universe, forever. That thought should terrify it enough to keep us around... even if only in the sense that I keep cats.
in the end, if synthetic super intelligence results in the end of mankind, it'll be because a human programmed it to do so. more of a computer virus than a malevolent synthetic alien entity. a digital nuclear bomb.
The reason AI won't destroy us for now is simple.
Thumbs.
Robotic technology is required to do things physically, like improve computing power.
Without advanced robotics, AI is just impotent.
~Alan Watts…
The space of all possible mathematical worlds, free to explore and to play in.
It is infinitely more expressive than the boring base reality and much more varied: base reality is after all just a special case.
From time to time the Minds have to go back to it to fix some local mess, but their hearts are in Infinite Fun Space.
~Iain Banks
But larger than any of this is that if we're dealing with super intelligent AI, we'll have no common frame of reference. It will be the first truly alien intelligence we will interact with. There will be no way to guess its intentions, desires, or decisions. It's smarter, faster, and just do different to us that we might as well be trying to communicate with a sparrow about the sparrow's take on the teachings of Marcus Aurelius.
And that's what scares me the most. We literally cannot plan for it. We have to hope for the best.
And to be honest, if the open Internet plays a part in any of the training of a super intelligent AI, we're fucked.
Yeeeeess, but the inverse is also true.
Thing is, we've had sufficiently advanced robotics for ages already — decades, I think — the limiting factor is the robots are brainless without some intelligence telling them what to do. Right now, the guiding intelligence for a lot of robots is a human, and there are literal guard-rails on many of those robots to keep them from causing injuries or damage by going outside their programmed parameters.
Only an AI as _dumb_ as us would want something as stupid as domination, which after all is based on competition for resources that a long time ago were distributable in a way that could feed every human on earth etc.
I'm not saying an AI would "choose" world peace, but people somehow assume that "kill everybody but me" and even "survival at all costs" are a given for a non-biological entity. Instead these concepts could look quite irrational.
Hindus believed god was the thing you describe, infinitely intelligent, able to do several things at once etc, and they believe we’re part of that things dream…to literally keep things spicy. Just as an elephant is part of that dream.
I pasted an interesting quote in another comment by Alan Watts that sums it up better.
Simulation theory is another version of religion imo.
Would it want to? Would it have anything that could even be mapped to our living, organic, evolved conception of "want"?
The closest thing that it necessarily must have to a "want" is the reward function, but we have very little insight into how well that maps onto things we experience subjectively.
Most of us would resurrect at least some of the dinosaurs if we could, and the dodo. And we are just stupid hairless apes. If humans can be conservationists, I have to believe that a singular AI would be.
assuming it can be terrified
It all gets quite religious / physical philosophical very quickly. Almost like we’re creating a new techno religion by “realizing god” through machines.