I'm not sure I understand how exactly a human-equivalent AGI would be more dangerous than an actual human.
* No improvement beyond human level (correlation between IQ and job-performance is variously estimated at around 0.5 and IQ largely comes down to executive function and working memory - so we have to assume an AI design that doesn't allow adding more working memory for some reason)
* No extra individual capabilities (eg human-level but with mental access to Google Scholar)
* No extra group capabilities (eg human-level but with telepathy between AIs - look at the difference the internet is making to human productivity eg Linux)
* No horizontal scaling over time (eg one human-level AI now, one million once Moores law kicks in - that's a lot of scientists/strategists/politicians/advertisers)
So in the situation where we make an AI that is somehow structurally limited to human-level intelligence and doesn't benefit from extra interfaces and we somehow can't afford to ever make many of them, then, sure, I can't think of a way that it could be dangerous.
In a few minutes of cursory thought I came up with four assumptions that were necessary for assuming that they won't be more capable than humans. What are the odds that all of those hold?
[EDIT isn't dangerous -> doesn't have the capability to be dangerous]
I think a more interesting question is "how do we make an intelligence, artificial or human, that isn't potentially dangerous?"
Humans have a bunch of low-level primate aggression that seems to be wired in; we can modulate it (we're largely self-domesticating) but we can't totally get rid of it and more importantly, we can't look at a baby's genome (and nurturing environment) and say "yup, this one's going to grow up to be Hitler", or "this 'un's going to be a pacifist, altruist, and general benefactor".
On the other hand we can probably add a bunch of monitoring code to any AI we develop rather than grow: if nothing else, consider the possibility of simulation runs looking for an empathy deficiency.
The OP said:
> I'm not sure I understand how exactly a human-equivalent AGI would be more dangerous than an actual human.
Since an actual human might be out to get me already the question is not motives but capability - should I fear evil AIs more than I fear evil humans? Or an evil human with a friendly leashed AI?
The argument typically revolves around whether or not AI would self-improve and lead to runaway intelligence, but I think that that's a distraction - there are plenty of other ways AI could be dangerous that are easier to persuade people with without running into the that-sounds-like-scifi filter.
We can't even get humans to agree on ethical values, let alone explain them to an AI, so even a perfectly altruistic AI could still be a disaster if it has the wrong value system, or if we don't perfectly understand the consequences of our own values systems when fully enacted.
If the A.I can self-improve, then we can also improve it. We're still equivalent.
Hitler had literally millions of human-equivalent-intelligences - they even had excellent sensor and actuator suites - and it didn't work out so well for him in any case.
Sure, you're right in that humans don't understand enough about our own brains to perform such modifications, but that may simply be down to a lack of knowledge, rather than not having sufficient cognitive ability to comprehend the brain.
Additionally the hardware which an AI can run on is generally assumed to be significantly more malleable than our limited gray matter confined within our cranium
The problem is there are hard physical limits you run into in computation - it's more efficient to just do the chemistry than it is to simulate it. Our meat brains might be more efficient in terms of density and power efficiency than any physically feasible digital computer.
Further, computing power can't be increased arbitrarily, especially when networks and switching come into play. Even if your simulated brain worked out how to improve itself, the resource cost to actually do so might be prohibitively high.
And on top of that, if they have human intelligence, they'd be able to improve it in those few seconds (because we had and they'd have access to our research), make smarter AI, and on and on exponentially fast.
I have a hunch that the constraint on switching speed actually results in intelligence rather than constrains it. Consider seizures in human brains for example.
Secondly, consider that humans have had the same brains for millennia and yet we still cannot improve them. We barely understand the first thing about them. I don't see how making the process faster, at the same level of cognition, would help.