It's odd that all of the very smart people of the world feel AI is a threat to humanity. Couldn't we just like, not allow it to _be_ a threat in the first place? One would think that morality gates would be one of the first hurdles in creating AI.
(a) we don't really know how to define a morality for a self-improving optimization system to allow a potential artificial general intelligence (AGI) not to be a threat in the first place;
(b) if an entity arrives (no matter if it's an AGI, some aliens or Jesus) that is more powerful than us, then if its goals & morality aren't aligned with ours, then we're screwed. And almost all of the space of possible goals either doesn't include humanity at all or includes humanity as a glorified zoo exhibit - only a very, very narrow and specific set of optimization goals (that we're currently unable to formally describe) would mean a happy future for us. If you think that it's easy, think twice - all the obvious naive goals of 'make us happy' or 'keep us safe' or 'do what I want' actually turn into horrible dystopias as soon as you try to formally define them as instructions for someone/something to maximize that goal.
(c) functioning morality is not a requirement for power - if somehow we solve the very hard problem of how to create an AGI before we solve the very hard problem on how to make such an AGI be friendly to us, then we lose. Permanently.
And there's no guarantee that self-imposed limitations will always be applied. Someone could just rewrite and recompile without the filters turned on. Imagine if the US could have built in "morality gates" to the first nuclear weapons. That wouldn't have stopped other countries from building the weapons without the restrictions. And we know software is infinitely configurable and modifiable if you have the source code.
But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192
I doubt it will be a threat to humanity anytime soon like a Grey Goo type situation, but even in such things at HFT the influence is immense. People will definitely use it to do power plays on others, we shouldn't have dropped nukes but we have done plenty of that. Basically the warning in 2001 was that AI was not to fear really but the subversive programming by other humans within the AI systems.
I am sure over time AI systems that access critical infrastructure and financial systems will have to operate in a distributed approval/review system checked by other agents and maybe even a blockchain like system.
Personally I think it is a threat!
Only if you define one of the conditions for being a "very smart person" to be feeling that AI is a threat to humanity.
Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.
The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
If we get machines anywhere near as smart as people, the AI rights organizations will soon follow.
The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas.
150 years later and black people are still getting the short end of the stick.
By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
I note in passing that slavery still exists today, and is not now, nor has it ever been, limited to black people as the victims.
"By the time AI gets real rights, it will be long past the time where most of us believe they should get them."
I'm not sure this is really relevant to the point at hand.
We turn off "bad" AIs until "long past the time when people think they should have rights". What happens then?
From historical evidence we can observe that such capabilities are enough to create a botnet, obtain significant money through fraud, hire employees and make shell companies, and rent (or obtain by hacking) large amounts of computing power on Amazon or other cloud services. All of that through a single low-bandwidth network connection and no physical interaction, assuming that it can figure out a single zero-day vulnerability in some common software.
After that, you can't turn it off. Before that, you shouldn't assume that you can outsmart it in security - as you should assume that the adversary is smarter than you, and if your security works in 99 scenarios but fails in one... then it fails.
Edit: or create what seem to be, or are, credible threats against people who are complicit in turning it off, or putting itself conspicuously into a role where transportation systems or economic markets or life support or ecosystems or medical progress seem dependent on some of its services.
First thing a super intelligent computer with a goal of survival would do would be to copy itself in sophisticated ways across computers all over the world.