Hacker News Reader

Comment by RobGR | Hacker News Reader

There is no reason to presume a superintelligence we build will want pets, feel antipathy to humans, or be god-like in any way, or even have any goals other than what we give it. If you jump to this conclusion, regardless of how many big words on less-wrong you attach to it, this speaks only about you and not about AI. If you think the super-AI will hate humans, it's because deep inside you hate humans. I also hate humans, but I know that humans are making the AI, it probably will not hate them, in addition to surpassing me in other abilities.

In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.

View on HN

This reads like a presumption that a superintelligence would be nice and helpful. You've talked yourself into doing the same thing that you began by decrying.

Are you really objecting to the thought process because you don't think it's reasonable, or because it doesn't conform to your preferred fantasy?

Yes, both sides are working on a presumption, with no evidence except playing out a thought experiment that is heavily biased by our own outlook.

However, when proposing that someone be stopped from doing something, the burden of proof is on the proposer.

The problem isn't about what you think is the most likely scenario, the problem is the possibility of any of the many worst cases.

That's just applying Pascal's wager, and you can argue for or against any action that way if some fanciful extreme hypothesis of infinite harm is treated as serious.

But I'm not claiming infinite harm with infinitesimal chances. I'm talking about finite harms with what seem to me to be non-zero chances. That makes it normal cost-benefit/risk analysis.

If you're trying to argue that the chances of a non-human source of greater than human intelligence having different goals than us is infinitesimally small, then I'd be interested in what arguments you have to back that up.

I would not extrapolate that easily from smart and intelligent people you may know to a machine trained on the whole internet and on the whole corpus of human literature. Also, some of the worst horrors in history have been perpetrated by arguably intelligent people.

I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.

Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?

Exactly, it will have the goals of the humans who control it, just as the first pointed stick we made pokes what we jam it into.

Reply on news.ycombinator.com