Comment by adamlett | Hacker News Reader

Comment by adamlett | Hacker News Reader

I liked the article, and agree with it mostly. Here's a couple of ideas I see again and again that bother me about the singularity theory and the hype in general about general AI:

* The idea of exponential growth, which seems like an important underpinning of the singularity theory. Nothing in nature as far as I am aware, grows exponentially. Some growth trajectories may for a time look exponential, but they always turn out to be the bottom of an s-curve. The idea that once we develop the first AI with superhuman intelligence, that it will necessarily beget one of even greater intelligence, is deeply flawed for this reason. It is analogous to predicting that Moore's law will continue forever, because ever faster and more capable computers will assist in the design the next generation of chips. At some point the laws of physics will contrain further advances, and we will encounter the upper half of the s-curve.

* The idea of AI in a box. It's the idea that anything we would call intelligence can evolve divorced from a high-bandwidth, real-time sensory apparatus, and divorced from a way to directly manipulate its environment.

* The idea that more intelligence always makes a significant difference. If we look at thinking disciplines where computers are already better than the best humans (chess, heads-up poker, recognizing certain kinds of objects etc.), the differences are small. If the best human's decisions are 99% optimal, say, a computer's may be 99.9% or 99.999% optimal. The point being that a computer can never makes decisions that are better than 100% optimal.

1. An AI might not need long exponential growth to make humans obsolete. Maybe the computing power of one data center is enough to outperform and dominate humans? Imagine if you had infinite patience, and experienced time 1000x slower than everyone else. That number might be in the millions. Or more.

2. I agree that it's a very strange and unpredictable scenario.

3. How optimal do you think humans are in the real world? In chess you have a very limited number of possible actions at each point, but in reality your possible moves are almost limitless.

1. I'm not sure what you mean by the "computing power of one data center". Surely not one present-day data center? And whether or not I can imagine experiencing time 1000x slower than everyone else, has no bearing on whether or not that's a realistic scenario. I can imagine all sorts of things, like Moore's law holding true forever, or P=NP.

3. That obviously depends on the endeavour. But humans are pretty great at a lot of stuff. We learn quick too.

> And whether or not I can imagine experiencing time 1000x slower than everyone else, has no bearing on whether or not that's a realistic scenario.

It is a staple of AGI speculation that a computer program with even near-human IQ would spark the singularity since, at least, the hardware running it could be improved so that the AGI would be able to perform person-months of cognitive labor in days. Since the first target of this labor would be improving the AGI program and hardware, compounding improvements are expected.

Yes, I get that. I just doubt whether it's physically possible. And like the author of the original story, I also doubt very much that cognitive labor is the constraint that limits human progress. To believe so is what he calls thinkism.

> I just doubt whether it's physically possible.

Biological brains achieve their ends with relatively low speeds and energies. Even simplistic substitution with equivalent electronic components would be hundreds of times faster, and I'm sure we'll do better than that. I don't see the difficulty in the conjecture.

> I also doubt very much that cognitive labor is the constraint that limits human progress

That statement is impossible to discuss without defining "human progress", but if the work of the world's universities for the next 100 years was available in one year's time, at the very least someone with access to that information would have a significant competitive advantage. It seems clear that the next 100 years will include significant advances in automated fabrication, at which point physical labor also essentially becomes cognitive labor.

Even simplistic substitution with equivalent electronic components would be hundreds of times faster, and I'm sure we'll do better than that

The way you state that one wonders why we haven't already achieved superhuman, general AI.

if the work of the world's universities for the next 100 years was available in one year's time

But that's a fundamental misunderstanding of the character of the work that goes on in universities. Knowledge is only rarely produced by people thinking about problems. It's produced by mundane trial and error experimentation. Experiments that take time. And money.

> The way you state that one wonders why we haven't already achieved superhuman, general AI.

Brain emulation is still expensive ...

https://en.wikipedia.org/wiki/Artificial_brain#/media/File:E...

It'll be interesting to see what we can learn at the different levels of emulation.

> It's produced by mundane trial and error experimentation.

Fair point, although experiments in computer science and applied mathematics can usually be carried out without constructing physical apparatus. Also identifying and designing experiments to efficiently characterize a problem space is a large part of experimentation. And again, once automated fabrication improves, some automated experimentation in physics and chemistry becomes possible.

"The idea of exponential growth, which seems like an important underpinning of the singularity theory. Nothing in nature as far as I am aware, grows exponentially."

Actually a lot of things grow exponentially, at least for a while. Consider zygotes, every cell division period they double.

That said, AI is exactly not "natural". A major point a lot of folks miss thinking about this is AI is amenable to engineering. This means that AIs can be intentionally designed to not have the limitations of natural systems. For instance, even today we know how to make effectively infallible, very fast memories of arbitrary size.

The other thing that makes strong AIs seem likely to improve exponentially is the idea that the first strong AIs could be focused on creating better strong AIs - and so on. Since AIs may also be replicated at will, a whole lot of distraction-free, singleminded intellect will be brought to bear on the next-gen AI design, and there is no reason that process couldn't be repeated indefinitely.

"At some point the laws of physics will contrain further advances, and we will encounter the upper half of the s-curve."

That might be true, however we also might be at IQ 1,000 (or IQ 1,000,000) AIs by then...

"The idea of AI in a box. It's the idea that anything we would call intelligence can evolve divorced from a high-bandwidth, real-time sensory apparatus, and divorced from a way to directly manipulate its environment."

This could be approached from many directions, from teleoperated robotics to VR.

"The idea that more intelligence always makes a significant difference. If we look at thinking disciplines where computers are already better than the best humans (chess, heads-up poker, recognizing certain kinds of objects etc.), the differences are small. If the best human's decisions are 99% optimal, say, a computer's may be 99.9% or 99.999% optimal. The point being that a computer can never makes decisions that are better than 100% optimal."

This seems a poor argument. I'd argue that greater intelligence introduces qualitative differences. For instance, the capabilities of a human are far beyond those of a dog. The same can be said of dog versus a grasshopper. Even the difference between an Albert Einstein and an average person seems profound.

I believe it's quite impossible for even an IQ 200 human to imagine the capabilities of an IQ 1,000 AI. What does a dog think about the reasoning capacity of a human?

Reply on news.ycombinator.com