The author finds it a readily apparent axiom that a vastly superior intellect cannot be controlled by a vastly inferior intellect.
Generally, that seems valid to me.
Generally, that seems valid to me.
I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce.
The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours.
Like under this assumption the AI agents behaviour is by actual definition unpredictable.
Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model.
If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve.
It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.
The mistaken assumption that GP is talking about is related to conflating intellectual capacity with physical capacity.
Another analogy is a brain in a vat. If the brain only has read capability over the world surrounding it, then no amount of processing power can enable it to change anything about its situation.
If you don't have a gun and a knife and you're on Vancouver Island, I assure you that the bears will eat you no matter how keyboard macho you think you are.
This is exactly the same thing a chess program does when it avoids your trap. Intelligence compensates for physical weakness.
Go to a zoo.
Are bears walking around with humans in cages?
People seem to think arguing that something is true by definition is some airtight case. If your definition of superintelligence includes a part about not being controllable by humans, then you have a weird definition and calling that thing "superintelligence" is an attempt to be confusing on purpose, or to smuggle your argument into the definition. Either way, it's not something that someone who is thinking carefully does.
Note: I am not taking a stance here on whether humans can or can't control a superintelligence. I am saying it is not true or false by definition, and in fact I think nobody knows for sure. It's probably true for some superintelligences, and false for others.
Parasites.
There are many creatures which lack any kind of trait we would call intelligence that can use vastly more intelligent and complex creatures to their own ends.
We are simply marginally more intelligent and that has resulted in massive gains.
An entity with an IQ of 1,000,000 after six hours of recursive self-improvement would be *vastly* more intelligent.
Evolution is much like that old sporting adage, "you can only beat what's in front of you". I think there are a lot of assumptions about the nature of evolution to assume that life wouldn't find a way.
Even saying that intelligence is an ability for pattern recognition but the patterns that you need to recognise to grow more intelligent are ever more complex and obscure. How much compute power is an IQ of a million? Where does it get its energy from? How does it process the complexity in the greatest patterns in a reasonable amount of time and space. What is going on that much faster than we experience time at that is worth perceiving?
B) prisoners regularly escape from physical systems. AI is already being connected to the internet to perform actions on the user behalf so not unreasonable to think that
C) look at ex machina - not an unrealistic scenario where the AI takes advantage of an opportunity to get out of jail
D) human physical systems have exploitable flaws. Physical systems are easier to secure than digital ones. Human software security mechanisms have flaws. How are you going to build this jail if the AI is just spending its time probing weaknesses in whatever system you’re running it on to escape the jail / sandbox?
Idiot Australians who smuggle drugs to/from Indonesia come to mind.
I don't agree, as that's highly dependent on that singular criminal and the resources police are willing to expend on catching them. The police definitely have an advantage, as I pointed (most recently) here[0]:
Law enforcement aren't superhuman. They're just as dumb (or smart, but the
really smart ones end up in corner offices rather than police stations like
police and more common criminals) as the next guy. Their big advantage,
especially in a circumstance like this, is that they only have to get it
right (i.e., find some evidence) once. The alleged perpetrator of a crime
needs to get it right (in covering their tracks, destroying evidence, etc.)
every single time to make sure they aren't identified and caught.
Edit: Corrected link. Clarified prose.
There are supposedly brilliant people serving life sentences right now and the only reason they're facing that situation is because they couldn't shut the fuck up when they sat down in a little room with an at best slightly above average intelligence police officer.
Stupid is as stupid does.
If I prepare for a test and perform better than someone smarter than me, my working knowledge made me superior at that task, even though I was less intelligent at the time I performed it.