Unfortunately, we didn’t learn from the proliferation of nuclear weapons:
“If we don’t make nuclear weapons, others will make them. If we make them then they will make them. If we make them and they make them, we’ll have to make more. If they make them and we make them, they’ll make more...”
Note: This fallacy also helps sell guns and martial arts lessons.
I don't see any reason to believe that intelligence forms an infinite ladder, I mean, it's fun to think about, but surely Zeno catches the tortoise eventually!
This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.
I realise that all of these arguments come down to "AI of the gaps". But in true Dunning-Kruger style, we don't know what we don't know - not just about AGI, but intelligence of any kind, including our own.
That's not D-K, but Rumsfeldian Uncertainty.
That idea even lines up with some crude experimental data -- vast additional resources barely make a dent in the frontier for chess, image recognition, a breadth of graph problems, etc....
It seems similarly ridiculous to imagine that the human brain just happens to be close to the maximum computational power because it was what we suppose to be the most powerful instrument produced thus far on our planet.
Wouldn't it be equal parts unlikely and strange if the most effective and compact process that can be produced in any conditions just happened to be the one that random evolution happened on in a random walk through what is practical with earths organic chemistry.
Even if we imagine the incredibly unlikely idea that you can't make a more effective mind you would think you could run the same process faster than real time or more instances of it for increasingly small resources on a better computer.
While we're on the topic though, no it doesn't have to be unlikely or strange for people to be near-supremal, and throwing more hardware at the problem doesn't necessarily make it go away:
On some level that idea comes down to defining what we mean by "intelligence." Suppose we can only measure intelligence by proxy by measuring performance on specific tasks. Then the composition of those tasks plays into any final intelligence score. If they're all fully parallelizable then you're totally right that people almost certainly aren't anywhere near the "top," and moreover a "top" probably wouldn't even exist. In standard parlance though, intelligence is something more than the ability to add a lot of numbers quickly, and most of our current measures of intelligence have exponentially (or worse) diminishing returns as more hardware is added.
In that latter case where intelligence is measured by performance on algorithms which are fundamentally hard, people wouldn't be near the top because of some rare process which accidentally made us that way. People would be near the top because _anything_ displaying a modicum of thought would be near the top because of the vastly diminishing returns of additional hardware. As a crude ballpark, if you mustered every atom on earth into its own processor running at 4GHz then you could solve a fully parallelizable exponential problem roughly 4x bigger than what one of Google's newest TPU's can manage.