This is pretty absurd to point to as an intermediate goalpost; that's basically game over when it happens.
This is pretty absurd to point to as an intermediate goalpost; that's basically game over when it happens.
As for it being 'game over', why? Is there something inherent in AI that would necessarily be inimical to human beings?
Because, the further part of the story goes, machines think quicker and they'll go on improving even quicker, all while having potentially different goals from us.
To be completely fair, the facts on the table are: 1) no one knows where the "intelligence ceiling" is, and 2) in many tasks where machines outperformed humans (image labeling, porn classification, speech-to-text, games like go or chess) they keep on improving, sometimes well beyond the human level.
One might have said the same thing about corporations in 1900...
The current population explosion is evidence for, not against, this, IMO... stars also enjoy boom times, even as they consume the last of their resources, in ever more exotic formulations...
At the very least, any team attempting to keep a strong AI in containment should have stringent background checks for all engineers connected to the project, and the engineers should be screened in a borderline discriminatory fashion. Only engineers with families should be allowed to get near the containment. Single/lonely engineers or engineers undergoing divorces should be kept away. Engineers with debt should be kept away. I'd even go as far as to say that engineers who enjoy science fiction media should be banned from the project. Ideally you'd bring on professional psychologists to create a set of criteria designed to minimize the possibility of an engineer deliberately breaking containment.
But frankly it just shouldn't be attempted in the first place.
rm -rf rogueAI/
If that doesn't work, there's always a circuit breaker.
Unless we are near some kind of physical upper limit on intelligence, any AGI we build will easily out smart us, probably in ways we can't even conceive of.
Years ago, an A.I. designed to become an oscillator, i.e. produce a sinusoidal wave, learned to be an amplifier instead, taking advantage of the 60Hz noise that it got from the power grid. Its makers had not seen that coming. And we're talking about a very dumb machine by general intelligence standards.
The fact that it gets 60Hz from the grid confounds the results and might not be meaningful. The AI could for all we know have an easier time with the more difficult task of designing an amplifier.