Comment by jnurmine | Hacker News Reader

Comment by jnurmine | Hacker News Reader

Singularity...

The whole concept of the Kurtzweilian "singularity" has always felt to me like a technology-flavored hand-wave of eschatological thought: AI designs AI in a recursive manner, science/technological progress (whatever it means in this context) leaps ahead at an ever-increasing rate, then we all go to heaven and/or the world ends because robots kill us all.

Some people are big fans of the singularity meme (using the word in the original meaning, not mockingly), and I respect that, but I have always felt aversion towards it.

I cannot quite put my finger on it, but I guess it is partially because of the idea of ever-increasing infinite technological progress. This can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources. Strong AI or not, unless we go fully into science fiction with pocket-sized Dyson spheres powered by quantum fluctuations and powering an entire planet and blah blah blah, such AI would have to interface with the physical world to do anything meaningful, and it would be limited by the available resources and most importantly by what is feasible and what is not.

Edit: fixed some of the typos

> ... then we all go to heaven and/or the world ends because robots kill us all.

Scary and/or overhyped language aside, I don't get why is this such a hard concept to grasp for generally educated people. Humans were a singularity from the perspective of chimps (and various other animals). It doesn't mean we immediately hunted down every chimp in existence, it means chimps became irrelevant and only a few of them remain - mainly in zoos and reservations. I would've called it a "phase transition" rather than a "singularity", but the main point is pretty clear. It's about a new adaptive system taking over the task of burning through local resources faster than its forerunner.

Eukaryotes were a phase transition (better biochemical specialization), Multi-cellular organisms were a phase transition (better biological / behavioral specialization), Mammals were a phase transition (energy regulation, better brains, social systems), Humans were a phase transition (general-purpose language & culture, scalable coordination), and now machines will be the next phase transition (shedding the biochemical shackles & incremental evolution, so to speak).

BTW, humans were a phase transition primarily for exhibiting Turing-complete structured communication and imagination (there's a thesis around this topic called the Romulus and Remus theory of prefrontal synthesis [1]). In a way, we're living in a world that has already gone through a recent singularity, and it was us who caused it.

[1] https://www.biorxiv.org/content/10.1101/166520v9.full

>his can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources.

I've always though of singularity as a sort of runaway progress scenario. Human brains run on a few hundred calories per day. I imagine if we could build an AGI, and even if it used 10,000x the power to run, we'd still be in a singularity type of new frontier. Imagine spinning up thousands of super brains at a time. I think it's hard to predict where that kind of capability would lead.

> The whole concept of the Kurtzweilian "singularity" has always felt to me like a technology-flavored hand-wave of eschatological thought

Kurzweil is explicitly counting on it for his personal immortality. That seems pretty eschatological to me.

> Kurzweil is explicitly counting on it for his personal immortality.

Is he still? At 74 years old that's verging on irrational.

He could live another 30 years. You never know.

Sure, obviously you can't defy physics just by getting smarter.

My parsing of the singularity is the point at which the speed of improvement exceeds our ability to understand it. "Predictability" is the usual term, but I think it introduces more problems than it solves.

The reality is that we humans have a finite rate for internalizing and utilizing knowledge. The singularity "happens" on the day artificial intelligence begins generating (and subsequently using) new knowledge faster than our human limit.

An issue with the Kurtzweilian "singularity" is that he didn't come up with the term - it was from a casual conversation of John von Neumann:

"... the accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue"

It's always bugged me a little that it isn't very defined. Personally I think of it as the point where robots can make better robots without needing humans.

I am not a believer in the imminency of "the singularity", but I think it would be worth considering that sentient AIs would likely be as immortal as their power source and compute substrates are, which would allow them to think and plan and operate on timescales that are utterly alien to us.

If one were to assume that they were substantially wiser than us, they would likely choose to put themselves on a path to expand beyond Earth and take advantage of the vast wealth of resources beyond. With any luck, they will decide that this messy biological planet is a bad place to be, and will leave us rather than destroy us ;)

> This can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources.

At some point, energy density would increase to the point where you’d cause an actual singularity via gravitational collapse.

Reply on news.ycombinator.com