Hacker News Reader

Comment by mitthrowaway2 | Hacker News Reader

I'm much more scared of the AGI, because even the most inhuman human is still human, and I can have sympathy for them.

Won't any AGI made by humans unavoidably have human origins, and thus, simply be a [post-]human? I don't see a reason I should be scared of a non-biological but generally human-like intelligence - not on a general principle. If anything, in general, I'd be happy that someone is finally getting real good chances of conquering death.

I possibly would fear genuine non-human intelligence (on a principle of fearing what we don't understand), but given that chances of one evolving under our nose, on its own, entirely within its own world, without the connection for ours, then emerging, are quite improbable... Unless real aliens arrive - but let's avoid spoilers for 2025 this early.

What should I fear, I think, is hostile intelligence, but then I'm not sure I should discriminate on its origin. I mean, there are some very real humans out there who can try to end this world (in one way or another), so I'm probably kind of desensitized and isn't exactly afraid of some very theoretical AGI threats.

If the AGI were made from a detailed scan of a human brain converted into a physical simulation, or otherwise proved very similar in terms of embodying human values, then perhaps. But it doesn't look like things are going this way. Humans make a lot of things that range from clever to mundane to horrifying, and I can't think of a single one of them that I would be happy with replacing humans across the surface of the Earth, even if it were very good at maximizing production figures of a paperclip factory or responding helpfully to queries.

That’s a hostile intelligence scenario, right?

What I wanted to say is that, I, in particular, really don’t care if it is a human trying to harm me, or a machine. Both of those scenarios are so unfavorable, I really don’t see a point in having any preference. So, I guess, I can’t think of any evil human to replace with even just an evil robot either.

But I think I’m less imaginative of a machine becoming sentient and hostile, than a human doing so. The former hadn’t happened - and isn’t exactly trivial, given how big this fear is a part of human culture, and how slow and limited first AGI is inevitably going to be. And then we have an abundance of the latter.

I think it's more like, even if a total psychopath took over the world and turned it into their personal dystopia, I think there would still be more human-goodness in that world than if a paperclip machine took over. Not merely more (nonzero) humans, but also more things that I and other humans value. Things like art, nature, beauty, play, and maybe hope. There would be some things that the psychopath dictator appreciates and values that I also appreciate and value, even if that person is entirely hostile to my existence. A future of silicon and steel, methodically trampling the earth and grinding the solar system into paperclips, simply doesn't contain any of that. So I prefer the dystopia on those grounds.

Does escaping death really sound good to you?

I mean I don't want to die before my time so to speak, but I'd say at some stage I'll have enough.

I'd say the best outcome would be having excellent medicine to ensure something like a relatively fit body for 1-200 years and then see how that goes first. Living forever, not having any kind of conclusion? Seems like a careful what you wish for.

Ah, sorry. “Conquering” as having full control over it, so it never comes on its own terms. Or as close to that as physically possible.

How do you know part of being self aware and intelligent also means compassion is part of the package ?

Sorry, I don't understand what you mean. Did you mean to reply to a different comment?

I mean, maybe what happens is, intelligence + self-awareness gives rise to compassion in even digital beings. Understanding might be the basis for compassion.

Yes, maybe what happens is exactly that. We can put on a blindfold, hit the gas pedal, and trust in the benevolence of nature that things will go our way.

But AI doomsayers aren't saying that there's no possible good outcomes, they're saying that the good outcomes are a narrow target surrounded by a vast ocean of bad outcomes, and it will take more than luck to hit them. It will take careful, detailed analysis, and a great deal of precaution, verification, engineering, and double- and triple-checking. Not luck. Not hope.

Have the doomsayers prevent anything significant from going forwards ?

While this idea sounds a bit fluffy, I actually think there may be something to this, understanding the “other” may very well be an integral component in what we define as sentience.

Sounds kinda like your problem ngl

Reply on news.ycombinator.com