Hacker News Reader

Comment by pavel_lishin | Hacker News Reader

I'm sorry for bringing this up, because it's vaguely controversial in some circles, and worse, a huge time-sink on par with TVTropes, but Yudkowsky attempted to run an experiment to see whether someone could be convinced by an AI, using nothing but text, to release the AI into the world. Apparently, he succeeded - his subject elected to release the fake AI.

The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?

View on HN

It's obvious that many subjects would release the AI, if only because they don't really know why it's in a box in the first place. Furthermore, the experiment grants AI capabilities it would probably not have, such as read/write access to its own source code. It assumes the first AI we box will be thousands of times smarter than we are, which is bizarre, since we should already have access to a whole continuum of AI dumber and smarter than us. It deprives the gatekeeper from a lot of tools they would probably have access to in a real situation, for instance raw computing power that far exceeds the AI's and runs powerful analytics on the AI's brain, or the ability to take snapshots of the AI's state and run many conversation paths.

The kind of AI you're describing is not the one anyone is worried about or that would be interesting in the first place; the issue is with AI that has access to its own soruce code (hence self-modifying and theoretically capable of getting smarter over time) and/or an AI that is smarter than humans.

What do you mean, it's not interesting? Human beings don't have unrestricted access to their own "source code", but they can still learn and get smarter. The AI I am describing can still be much, much smarter than a human being.

We're also assuming that an AI that can "modify its own source code" is going to do significantly better as a result, but that's not obviously true. If you had full access to every single neuron in your brain, you would have a lot of trouble figuring out how it works, much less how to improve it. Human-level AI would likely run into the same problem: the complexity of its code is greater than what it is capable of understanding. As for an AI that's a thousand smarter than a human, well, it's a thousand times more complex, still over the threshold.

I'm not saying that they wouldn't eventually figure out some improvements, but I suspect that it would end up being more trouble than it's worth. I think the evolution of intelligence is in fact the evolution of intelligence-producing algorithms. Any such algorithm eventually gets stuck in a local maximum, at which point greater intelligence requires restarting from scratch.

> Apparently, he succeeded - his subject elected to release the fake AI.

Hasn't he refused to release the transcripts for this experiment? And I think there've been subsequent runs of the experiment, with some losses as well as wins.

Edit: I see that not releasing the transcripts is part of the design of the experiment. Which seems kind of like it'd seriously hamper any kind of independent analysis of the results or the quality of the experiment.

That hampering is necessary to ensure that the fake AI is artificially more intelligent. If you allow retrospective analysis, then you can reasonably outthink the AI because you have more time to evaluate and make the decision.

With a real AI, you will not be able to do this: the decision will already have been made and the consequences of its release already in effect.

Besides, you would need to repeat the experiment many times to get any meaningful data.

But this is all extremely hypothetical. Nobody knows what the real scenario will be, if something like this ever comes up - it probably won't be one person sitting in a room with a chat terminal and a "Release" button, under pressure to make a decision on the spot.

This contrived experiment is being done in the name of research, ostensibly to arm humans with the tools we need to deal with this situation if it arises. If the AI's reasoning can't hold up to a wider analysis, then you can only conclude that it's a trick of some sort, and hopefully when the time comes we won't rely simply on the testimony of one tricked person to make such a critical decision.

I have a lot of respect for Yudkowski but it honestly baffles me that he thinks this particular exercise should be given any weight whatsoever. It's totally unscientific, relies entirely on the unsubstantiated testimony of one individual, and if he weren't the one holding the keys I have to feel like he would be highly critical of it as well.

That exercise was not research, it was a counterexample to a single specific security claim: that we can "choose not to release the AI," so we don't have to worry about them being dangerous.

You don't need to research a counter-example, you just produce one. He did so by exhibiting a relatively weak AI that couldn't be contained.

>You don't need to research a counter-example, you just produce one. He did so by exhibiting a relatively weak AI that couldn't be contained.

Well, no. He showed that a person pretending to be an AI could convince a person to let it out of the box. Sometimes.

It's not really a meaningful counterexample. Everyone involved with the event knew that the AI wasn't real and that there are zero consequences for letting it out. You'd have to concoct a much more involved test as a decent counterexample.

A person pretending to be an AI is surely easier to contain than a super-powerful AI that is smarter than all people. A failure to contain the former -- a strictly easier task -- is a perfect example of a failure to contain the latter.

> how do you trap and imprison something smarter than you?

Jails seem to have this figured out just fine.

The correct analogy would be a jail run by a bacterium.

Prison bars are dumber than a bacterium and they work just fine.

Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.

The problem isn't with AIs dumber than people - it's with what happens when someone finally builds a smarter one. It doesn't matter how good are prison bars if you're smart enough to social-engineer your way out of the cell. And human criminals do escape from prisons every now and then.

This is more like a 'reverse hacker', instead of a brilliant hacker trying to get into a system it is a brilliant hacker trying to get out of a system, and in this case the hacker is likely vastly more brilliant than the defenders. The same rules apply: the hacker has to succeed only once, the 'jailers' have to succeed all the times. Predicted long term outcome: escape.

Your moving the goal posts.

Can I keep an intelegent AI in a box. Shure, unplug it.

Can I keep all AI's in a box well no.

PS: Lot's of dumb things are said about AI's. Sadly, people tend to think in terms of Science fiction as Magic but in The Future. And then picture AI's as the ultimate wizards able to reshape reality to their whim. Reolistically the first true AI may find programming boring and so much for the singularity. If AI smarter than you is a bad idea it's unlikely for a progression of AI to keep building ever more intelegent replacements.

> Your

> intelegent

> Shure

> Reolistically

Spell checker broken?

Anyway, you're going to have to develop your AI somewhere, you're going to have to move it to the box somehow you're going to have to train it somehow and you're going to have to have it interact with the real world somehow. All of those are opportunities for escape, I think 'unplug it' sort of defeats the purpose of having an AI.

What if you assume that the AI might be 100x or 1000x smarter than its guards, not just 2x or 3x smarter?

And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned

> What if you assume that the AI might be 100x or 1000x smarter than its guards, not just 2x or 3x smarter?

If you can make an AI that's 100x smarter than you are, surely you can make one that's only 2x or 3x smarter. Why not start with the latter? If the experiment turns out well, then you can use the 2x/3x smarter AI to guard a 10x smarter AI, and so on.

> And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned

Most of the time, it is much harder to come up with solutions to a problem than to verify that the solutions are correct. If you consider NP problems, for instance, we can use the AI to find good solutions heuristically and we can prove the correctness of their solutions with a trivial algorithm. That would still be very useful, but there wouldn't be any room for foul play.

It seems it would be very hard to place an arbitrary restriction like that (2x human intelligence vs 100x human intelligence) on something that is learning. I think the idea here is that the AI would begin to learn on its own and become smarter at a frightening pace, not that we would set out to create something that had exactly 2x human intelligence.

How do you quantify human intelligence like that anyway? We are very similar to proposed AI/machine learning techniques in that we form predictive models from observation, but how can that really be measured?

> I think the idea here is that the AI would begin to learn on its own and become smarter at a frightening pace

With what resources? The AI is in a box, remember: we are running it on a machine with only so much memory and processing power. There's only so much it can do within these limitations, it's not like we're giving it a spending account.

Reply on news.ycombinator.com