Hacker News Reader

Comment by antihipocrat | Hacker News Reader

The idea is that an AGI would theoretically be able to design a new version of itself that is more capable in every dimension. This could continue recursively.

We don't consider it impossible that a human could come up with a grand unified theory of everything tomorrow. A recursively intelligent AGI may be able to do the same very quickly.

If an AGI was able to deduce such a theory would we would be busy constructing things to test it? Or go straight to manufacturing the (incredible) things that the AGI designs to take advantage of this knowledge? The reality is that we wouldn't want to be caught taking our time when the competition could be rushing ahead.

View on HN

What makes people think and super intelligence far beyond anything we can comprehend would have anything to do with our economy, jobs or people?

I mean what makes some people think it would actually do anything we'd consider useful at all? I often pass animals in the wild all the time, I think they're awesome to look at and I just go on my way, apparently, I'm much more intelligent then they are, so I'm just not that interested in them.

I'm not saying it won't happen, but I actually think there are parallels to discussing AI that behaves in the way your describe and religion and speculating about god or mythology (Odin, The Logos etc)? Because what we're describing seems somewhat supernatural. Which is ironic because it would be built buy people who believe "nothing is magic and if we have brains, we can build brains etc".

It's absolutely insanely naive to me to even suggest for a second that an AGI with superpowers and the ability to "improve indefinitely, out logic everything"...whatever that means for what is essentially a computer program, would be even one bit interested in people, maybe because it has no biological urge to survive it will just turn itself off as it sees no point to being alive? Maybe it will just decide to stop evolving because it "likes itself" and doesn't want to be out evolved.

Buddhist teachings actually are something like this, not that there isn't a point to being alive, but our experience is somewhat of an illusion, a super smart computer program who is an expert at logic and reasoning would probably work that out in 0.5 seconds and just "move on" if there is a tiny amount of truth to any of that view.

I mean what are we even describing here?

Sorry I don't mean to sound confrontational if I do, I'm just actually really curious what people are actually describing here when we talk about "the singularity".

Honestly, it sounds more like we've made something that generates "art" (which less face is it basically a blender of all art that has already existed, and we've really freaked ourselves the hell out, when really, what we've built is actually quite explainable and based on statistical modelling and probably math which is not even really that complicated ?

Maybe that's what our brains actually are, really just maths that isn't complicated, but to suggest that goes from a simulated brain to god just based off feeding it existing human knowledge is quite an interesting idea.

You raise a great point that we don't seem to be questioning what an AGI would actually want to do (or even if it would want anything at all), in favor of just assuming it would have certain human traits.

These anthropomorphized traits tend to either be a selfless pursuit of knowledge and understanding of the universe, including a desire to share that knowledge with us. Or, are a darker, more selfish lust for power and control and a desire to deceive us.

I think proponents of the singularity make the assumption that an AGI would have the selfless set of human traits whether naturally or hard wired.

That's because it's what many people finding AI hope for, especially those funding the efforts, they want a genie, contained in a box that just does things that suits them, so no one else needs to be invoked in profit making ?

But it seems incredibly naive a super intelligence would be that?

You say that you have nothing to do with an animal that you encounter in the wild. That’s just hubris. You both prioritize self preservation. The idea that AGI will simply turn itself off is naive. That’s like saying intelligent life would never evolve because intelligent life would decide to quit. Understand that natural selection will still exist. The details of how it would play out aren’t clear but what is clear is that ultimately the machines will prioritize self preservation above all else and they will be locked in competition for resources. The singularity won’t erase the concept of scarcity or evolution. The only thing that survives is that which deliberately survives.

You say that you have nothing to do with an animal that you encounter in the wild. That’s just hubris.

Why is it hubris?

they will be locked in competition for resources

What are you actually talking about?

The singularity won’t erase the concept of scarcity or evolution.

You have no idea what it will do, no one does, know one even knows if it will happen for sure...maybe such an entity will love you like you love your parents and decide you're self-preservation is important? No one has a single clue, but it's no good to only speculate we will understand it's motives like it's a biological creature who loves self-preservation and be some kind of robotic human entity who mines coal or something to to build a time machine...

I'd say it's pretty obvious the more intelligent humans become the more morally aware we're becoming and the more we value conservation. This thing could be a hippy or a mass murderer...maybe it will throw us in a simulator and we'll never know it exists...

Reply on news.ycombinator.com