Hacker News Reader

Comment by neonbat | Hacker News Reader

Oates' argument generally seems to be "It won't be scary because it won't be." He rejects the premise of Musk, Gates, etc. He basically says we won't be able to build a superhuman AI. All his arguments are then predicated in this assumption. The assumption that we won't build a superhuman AI is just wrong.

For example:

> [Would this superhuman intelligence inherently go nuclear, or would it likely just slack off a little at work or, in extreme cases, compose rap music in Latin?]

This statement assumes an AI on the caliber of a human intelligence. Making rap music in Latin is something a human would do. A superhuman AI is beyond a human intelligence in the same way a human's capacity for thought and tasks is beyond that on an ant.

The premise of a superhuman AI has at least some evidence supporting it. Oates offers no solid evidence to his rejection of the premise. His argument is therefore kinda bad (but not necessarily incorrect).

View on HN

No, his argument is "nothing we have built so far (i.e. special purpose AI) leads us to believe that we will build general purpose AI".

There is nothing inherently contradictory about a future with ONLY special purpose AI. Technologically speaking, it doesn't really follow that one leads to the other. He is using his credibility as an AI practitioner to state this, although he doesn't go into details.

He's saying the burden of proof is on the doomsayers, which is reasonable. I still don't see the argument either.

The problem with that is that having proof and getting overpowered by a superhuman AI might fall within a very short timeframe.

The risk (impact*probability) is simply too high to be ignored. We don't know the probability very well, but considering the impact, it doesn't matter that much. I think if the risk is high enough, the burden of proof gets shifted.

To just swat it away with a sentence like "Sounds reasonable to you? Me either." is a bit insulting and reckless.

Personally I want to see AI get developed as fast as possible, but I think it's not too much to ask to care a little about the trajectory of that development in a phase where we still know we're in control. All this article told me is that yes, we're in control for now.

The problem with that is that having proof and getting overpowered by a superhuman AI might fall within a very short timeframe.

To be brutally honest, I don't actually see this as being necessarily a bad thing --- it's not like humans are doing such a great job of running the world. If a hypothetical AI is so much better at it than we are that it can replace us, then don't we deserve to be replaced?

Although I would make sure than anywhere halfway competent AI has the collected works of Iain M. Banks on its reading list.

It depends on how you define "better". As a sibling comment points out: let's say that the AI is programmed to maximize the number of paperclips in the universe. And then to accomplish that goal, it kills all humans (by eating our infrastructure to turn it into paperclips). But that's not enough. The earth's mantle and core contain massive amounts of iron and aluminum that could be used to make paperclips. So the AI starts consuming the Earth. And then it scans the rest of the solar system... ah, Mars and the asteroid belt contain lots of iron and aluminum too!

I fail to see how a solar system turned into a ring of paperclips orbiting a sun is in any way "better" than what we have right now.

The premise of human being replaced is all well and good, but it seems like you're assuming the superintelligence will do a better job. A paperclip maximizer may be able to deconstruct us on a molecular level, but that's not a thing for our light cone.

Superintelligence by definition does a better job. Think of it this way:

You're a human; how good are you at walking from point A to point B? You're actually quite good at it. You open up google maps (a human invention) and walk to point B. It only takes a small number of humans to maintain the infrastructure for you to find point B.

Now imagine you are an ant. How good are you at finding point B? You're absolutely crap at it. You need to organize an entire complex system of pheromones and ant-conga lines just to find point B (usually a food location). Most of your colony is devoted to this organizational task of just finding point B from point A.

That task that is all consuming for the ant and takes the full attention of the colony is peanuts to us humans, it's so freakin' easy.

By definition a super intelligence eclipses our conception of the universe. If you took the humans who had to try really hard at X, Y, or Z tasks, these X, Y, Z tasks would be peanuts to the AI, they would be so easy the AI doesn't even think about them. The same way ants try really hard and organize their entire colony to find point B and humans don't even think about finding point B.

There is Orthogonality Thesis [0] that says that (tl;dr) intelligence is orthogonal to goal system. So an agent may be very intelligent (be a powerful optimizer) but this doesn't imply it's goals will be "very good" by our (human) standard. You can have supersmart human-values-promoting AI, and then also a supersmart paperclip maximizer, just like you can have dumb versions of both.

The core of the "danger of AI" thesis is that any superintelligence that doesn't have its value system perfectly aligned with ours is a serious danger that will likely kill us all.

[0] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis

I agree; that is part of his argument. Obviously though it is a completely flawed argument. Up until the creation of the personal computer most machines were specialists. They were large computers designed for specific computation tasks or machines made specifically for manufacturing certain kinds of goods. Generalist computers didn't have much precedent. That's kind of the point of new technology. It's not necessarily predicated by something that came before. It is orders of magnitude better.

To me this argument has historically less validity than creative people dreaming about singular, binary acts of creation. Historically those people have won.

Up until the creation of the personal computer most machines were specialists. They were large computers designed for specific computation tasks or machines made specifically for manufacturing certain kinds of goods.

Sorry, but that's just not true. Back in the 60s, way before personal computers, the were many time-sharing companies, selling general "computing resources" to different people who would dial-up into the machines and run their own programs.

Yeah I thought about that when I was writing the reply. I don't really consider that a generalized use case. Is it more generalized? Yes. Computing itself, however, can be argued to be a specialized task. My computer today doesn't do "computing" it lets me read news stories, it lets me talk on skype, it lets me send mail, it lets me have philosophical discussions on hackernews. The personal computer is so clearly a generalized machine whereas those time sharing rigs back in the 60s were clearly not. They were specialized tools for programmers.

PLATO had graphical terminals with touch screens and software implementing forums, message boards, online testing, e-mail, chat rooms, picture languages, instant messaging, remote screen sharing, and multiplayer games. They weren't just used by programmers by a long shot.

Ok, fine. PLATO was a super cool more generalist machine that happens to not be a personal computer. My use of personal computers was just an example that I thought would get a point across about the evolution of technology. The fact that PLATO existed and was very different than what came before it (as per all the cool stuff you listed) if anything strengthens the argument I was making. Generalist machines were a big step up from what came before, orders of magnitude better. A super-AI will also be unpredictably orders of magnitude better than what we've done so far.

But my point is that the generalist computers weren't a sudden or unpredictable step, they were the result of slow and progressive improvement. If generalist computers are any example to go by, the fact that we don't have anything close to a primitive general AI indicates we're far, far away from developing anything close to a "super-AI".

That's not an argument why special purpose AI leads to general purpose AI with motivation take over the world, and I still haven't seen one.

The argument is that you don't need strong AI for it to be dangerous to humans.

I don't follow. See my top level comment about the "paper clip maximizer". The argument does hinge on strong AI, as far as I can tell.

Not sure I understand what you mean.

If humans who don't evolve anywhere even closely at the phase of technology and humans are more and more depending on machines (coinsidence?). If machines keep improving if software keeps improving, if technology become more and more networked then why is it on the doomsayers to prove any more than the trend they are claiming can have consequences that we will have to control or guide before they happen or it will be too late.

"There will be consequences" is not a prediction, it's a empty statement. Every possible course of actions has consequences. People usually claim more specific things, which is what the author is disagreeing with.

I wasn't trying to make a prediction and that wasn't what I was saying or the gist of my argument. The important part is the trend.

you are welcome to believe it can't end badly, I don't. In fact looking at millions of years of evolution I see nothing whatsoever that would make humans sacred from being outcompeted.

Reply on news.ycombinator.com