Comment by dojomouse | Hacker News Reader

Comment by dojomouse | Hacker News Reader

Interesting implications (which I'm sure people who've been thinking about it for longer than me have already considered) for the development and evolution of Friendly Superintelligent AI.

The way this is presented, personhood (which involves behaviours that we'd like a friendly AI to display) is an adaptation based on mutual benefit. We're not nice to other people because kindness and empathy is an unavoidable consequence of intelligence - rather we're nice because we're better off in a society in which we're known to be nice. And that's because we've got fairly limited personal capabilities so are heavily dependent on efficient cooperation with other people.

An intelligent entity without this constraint (e.g. a superintelligence that had access to advanced nanotech and could do pretty much whatever it wanted) w.r.t interactions with humans might not (probably wouldn't?) have any evolutionary pressure towards achieving and maintaining personhood. Which would mean we shouldn't take the approach of "Let's just make an advanced intelligence, because its intelligence will surely mean it behaves with empathy towards us".

"a superintelligence that had access to advanced nanotech and could do pretty much whatever it wanted"

Without water I will die in about 3 days. I have no "want" for the nicely brewed tea currently steeping on my desk. The point I'm making is wants invariably gravitate to what you can't get, at least not easily.

Its quite possible a nanotech equipped superintelligence (human or artificial) might really desire fame, or founding a new religion or philosophical school, or military conquest, or scientific discovery, or the creation of some really great art (fine or pop). Or maybe just collecting a vast pile of money. Those all have at least some requirement for empathy, however little.

Previous human fiction and semi-fiction along the lines of vampires, mythology, religion, and even robinson crusoe epics always seems to really like the idea of human culture even if the protagonist spends most of its time separate. As far as I know no one has explored the theme of introducing an AI to something like Buddhist meditation and letting it burn some machine cycles meditating, at least not in a hard sci fi setting (I would not be interested in that in a fantasy or soft sci fi setting, so if it exists I wouldn't know)

Some of this is self selecting in that its not hard to find lifestyles that human higher end intelligences like, and they're all vaguely empathic, and variations have only been cultural. Thomas Jefferson might not have empathized with his slaves very much, but that lack was because none of his people did, not because he specifically was very smart. There is probably some self selection in that smart people who completely separate from civilization are going to be unknown to civilization, so whatever they do, we as a group donno, so if they exist, whatever they do, there's not much evidence, and its likely an AI would have similar results.

Evolutionary pressure isn't going to be relevant here.

But our sample size of what consequences intelligence has is effectively 1. Drawing any conclusions from that is dubious. We just don't know.

However, that also means that the worst case scenarios we come up with are as well shaped by our idea what a human with unlimited resources might do.

I think there's a very good chance that the real outcome will be so far outside our sphere of imagination that it simply won't affect us much.

Reply on news.ycombinator.com