Hacker News Reader

Comment by Netcob | Hacker News Reader

The problem with that is that having proof and getting overpowered by a superhuman AI might fall within a very short timeframe.

The risk (impact*probability) is simply too high to be ignored. We don't know the probability very well, but considering the impact, it doesn't matter that much. I think if the risk is high enough, the burden of proof gets shifted.

To just swat it away with a sentence like "Sounds reasonable to you? Me either." is a bit insulting and reckless.

Personally I want to see AI get developed as fast as possible, but I think it's not too much to ask to care a little about the trajectory of that development in a phase where we still know we're in control. All this article told me is that yes, we're in control for now.

View on HN

The problem with that is that having proof and getting overpowered by a superhuman AI might fall within a very short timeframe.

To be brutally honest, I don't actually see this as being necessarily a bad thing --- it's not like humans are doing such a great job of running the world. If a hypothetical AI is so much better at it than we are that it can replace us, then don't we deserve to be replaced?

Although I would make sure than anywhere halfway competent AI has the collected works of Iain M. Banks on its reading list.

The premise of human being replaced is all well and good, but it seems like you're assuming the superintelligence will do a better job. A paperclip maximizer may be able to deconstruct us on a molecular level, but that's not a thing for our light cone.

Superintelligence by definition does a better job. Think of it this way:

You're a human; how good are you at walking from point A to point B? You're actually quite good at it. You open up google maps (a human invention) and walk to point B. It only takes a small number of humans to maintain the infrastructure for you to find point B.

Now imagine you are an ant. How good are you at finding point B? You're absolutely crap at it. You need to organize an entire complex system of pheromones and ant-conga lines just to find point B (usually a food location). Most of your colony is devoted to this organizational task of just finding point B from point A.

That task that is all consuming for the ant and takes the full attention of the colony is peanuts to us humans, it's so freakin' easy.

By definition a super intelligence eclipses our conception of the universe. If you took the humans who had to try really hard at X, Y, or Z tasks, these X, Y, Z tasks would be peanuts to the AI, they would be so easy the AI doesn't even think about them. The same way ants try really hard and organize their entire colony to find point B and humans don't even think about finding point B.

There is Orthogonality Thesis [0] that says that (tl;dr) intelligence is orthogonal to goal system. So an agent may be very intelligent (be a powerful optimizer) but this doesn't imply it's goals will be "very good" by our (human) standard. You can have supersmart human-values-promoting AI, and then also a supersmart paperclip maximizer, just like you can have dumb versions of both.

The core of the "danger of AI" thesis is that any superintelligence that doesn't have its value system perfectly aligned with ours is a serious danger that will likely kill us all.

[0] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis

It depends on how you define "better". As a sibling comment points out: let's say that the AI is programmed to maximize the number of paperclips in the universe. And then to accomplish that goal, it kills all humans (by eating our infrastructure to turn it into paperclips). But that's not enough. The earth's mantle and core contain massive amounts of iron and aluminum that could be used to make paperclips. So the AI starts consuming the Earth. And then it scans the rest of the solar system... ah, Mars and the asteroid belt contain lots of iron and aluminum too!

I fail to see how a solar system turned into a ring of paperclips orbiting a sun is in any way "better" than what we have right now.

Reply on news.ycombinator.com