Comment by drdaeman | Hacker News Reader

Comment by drdaeman | Hacker News Reader

Parent Full thread

Won't any AGI made by humans unavoidably have human origins, and thus, simply be a [post-]human? I don't see a reason I should be scared of a non-biological but generally human-like intelligence - not on a general principle. If anything, in general, I'd be happy that someone is finally getting real good chances of conquering death.

I possibly would fear genuine non-human intelligence (on a principle of fearing what we don't understand), but given that chances of one evolving under our nose, on its own, entirely within its own world, without the connection for ours, then emerging, are quite improbable... Unless real aliens arrive - but let's avoid spoilers for 2025 this early.

What should I fear, I think, is hostile intelligence, but then I'm not sure I should discriminate on its origin. I mean, there are some very real humans out there who can try to end this world (in one way or another), so I'm probably kind of desensitized and isn't exactly afraid of some very theoretical AGI threats.

If the AGI were made from a detailed scan of a human brain converted into a physical simulation, or otherwise proved very similar in terms of embodying human values, then perhaps. But it doesn't look like things are going this way. Humans make a lot of things that range from clever to mundane to horrifying, and I can't think of a single one of them that I would be happy with replacing humans across the surface of the Earth, even if it were very good at maximizing production figures of a paperclip factory or responding helpfully to queries.

That’s a hostile intelligence scenario, right?

What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either.

But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.

I think it's more like, even if a total psychopath took over the world and turned it into their personal dystopia, I think there would still be more human-goodness in that world than if a paperclip machine took over. Not merely more (nonzero) humans, but also more things that I and other humans value. Things like art, nature, beauty, play, and maybe hope. There would be some things that the psychopath dictator appreciates and values that I also appreciate and value, even if that person is entirely hostile to my existence. A future of silicon and steel, methodically trampling the earth and grinding the solar system into paperclips, simply doesn't contain any of that. So I prefer the dystopia on those grounds.

Does escaping death really sound good to you?

I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough.

I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.

Ah, sorry. “Conquering” as having full control over it, so it never comes on its own terms. Or as close to that as physically possible.

Reply on news.ycombinator.com