Hacker News Reader

Comment by blincoln | Hacker News Reader

I'm with you. If it were possible to create apocalyptic doom AGI 150 years after inventing electronics, by basically making a very fancy, procedurally-generated Eliza, then the universe should be full of Berserkers.

With septillions of star systems in the universe, the odds of us being the first species in 14 billion years to invent electronics seem remote.

Apparently even humans can figure out how to colonize the Milky Way in 90 million years[1]. A superior AGI produced by a "dark singularity" computing event should be able to do even better, but even so, plenty of time for some other species to have made a giga-Eliza that somehow became Skynet. Anything with enough self-preservation and paranoia to wipe out the species that created it would surely take to the stars for more resources and self-redundancy sooner or later.

[1] https://www.extremetech.com/extreme/294051-scientists-simula...

View on HN

An AGI being able to wipe out humanity doesn't necessarily mean that it can take over the universe. The world's governments are already capable of causing extreme suffering through a nuclear war. AGI risk scenarios aren't equivalent to an unbounded intelligence explosion. An AGI only needs to be more powerful than humanity to be a threat. It can be a threat even if it isn't that intelligent, as long as it gives unprecedented power to a few individuals or governments.

Both humanity and a super-intelligent AGI are bound by the laws of physics. Super-intelligence does not imply omnipotence; it simply means that the AGI is orders of magnitude more intelligent than humans. If humans can figure out how to colonize the Milky Way in 90 million years, then the answer to the question of why no AGI has done it is the same as the answer to the question of why no extraterrestrial species has done it.

This makes a lot of assumptions. Space is ridiculously big, and rather hostile to life, even artificial life.

You first have to survive long enough to become advanced enough to make electronics. You then have to not kill yourselves with nuclear weapons, climate change, or similar inadvertent effects of a rapidly industrializing civilization.

The planet and the solar system have to be friendly enough to space exploration and travel. Maybe there’s no gas giants for gravitational slingshots, or maybe no other rocky planets or an asteroid belt for mining materials.

Maybe the planet evolved complex life in extreme conditions, with such a deep cloud cover there’s no concept of outer space, so as far as the AI knows it’s conquered all there is.

Maybe the AI conquered the planet, but oops, there goes a super volcano or an asteroid and it gets wiped out.

And again… space is really really big. The AI may be on its way and just hasn’t gotten here yet.

There’s plenty of reasons why a super AI wouldn’t be able to conquer the galaxy and beyond, or why we haven’t noticed yet.

As I see it, the existence of AI is proof we really are alone out here.

Reply on news.ycombinator.com