Hacker News Reader

> The only thing worse than the "evil" genie giving us exactly what we ask for is the genie that decides its inferences about our actual intents/wants/needs/motivations based purely on our behaviors matter more than what we intentionally express as our intents/wants/needs/motivations.

It's not worse.

> a date rapist

A rapist shares 99+% of definitions and values with you as a fellow human being. AI won't (unless you somehow program them in). Rapist has his evil motivations, but won't suddenly invent a virus that kills the whole human species because you asked him to stop neighbor kids trampling your garden. AI might. If you tell it not to kill anybody it might destroy our civilization to prevent us from killing ourselves with global warming. Why not - it only makes sense.

Simplest way to ensure safety for the maximum number of human beings is to anesthetize everybody and put them on life support till their natural death. Perfect record is possible - you might cure addicts and prevent crimes and wars. If you specify people have to be awake as often as they usually are - it can keep them awake but restrained. You want people to have freedom of movement? Ok - you just released the whole prison population :) Keeping criminals in prison is OK? Then it may make everybody a criminal for a quick fix. Or just drug everybody to WANT to be restrained. And so on, and so on.

There's infinite number of possible courses of action that we discard without consciously thinking about them because of our assumptions. You have to put these assumptions in the AI, each and every one of them, and they are very subtle and invisible for us most of the time. And they often border on philosophy and morality, and defining them is political by definition.

It's probably impossible to code all our values and assumptions in by hand. That's a much bigger problem for safe general AI than a post-factum explanations that rub your morality the wrong way.

> Asimov's Three Laws

Are self-contradictory and useless for anything except literature.

And a drug addict, by analysis of behavior, is committing suicide so the AI should help them achieve their inferred goal based on that behavior. Nevermind asking them if that is their intent, it knows better because it sees the behavior.

Russell's laws encode NO limits and doesn't even demand the AI check its judgment with the so-called beneficiaries of its decisions. That is most assuredly worse.

Decoding the preferences of people from their behavior is just a first step, it's not the only rule, it's a rule that makes writing other rules safer than they would be otherwise.

I don't think that "sharing 99+% of definitions and values with us as a fellow human being" has really provided much protection against deliberate human-caused disasters. It's true that I wouldn't expect most people to invent a virus as a weapon, but I absolutely would expect governments to do so, and in fact I think they have done so.

> I don't think that "sharing 99+% of definitions and values with us as a fellow human being" has really provided much protection against deliberate human-caused disasters.

There was this thing called "cold war".

Tens of millions of people were killed in Cold War violence. It was a huge disaster caused intentionally by “fellow humans.”

I have issues with that estimate, but even accepting it - it was still a big improvement over WW2, WW1, colonization, Mongol invasions etc. And certainly preferable to a global thermonuclear war which was a real possibility.

Have you heard of https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...

Humans very deliberately created nuclear weapons, with the intention of using them to kill enormous numbers of people, which they did. The fact that humans created something horrible and then used that thing fewer times than they could have is hardly a triumph of humans, and it certainly isn’t great evidence to support the argument that humans are limited in their capacity for doing bad things because they supposedly share values with other humans.

Automated systems told them to start a nuclear war and humans didn't.

I don’t think that system was anything close to what would be described as an AI. Wasn’t it just a radar system intended to identify ICBMs?

Think of the system as (human + the warning computers + radars).

Thanks to the human element the system has shared human values and decided not to start a thermonuclear war despite it being the recommended course of action.

If it was a completely AI system it would probably just start the war.

But that system was designed (by humans) to give information and warnings to a human. Of course it would be a mistake to design a system like that and then remove the human.

Perhaps a less controversial example would be the interpretation of the three laws by the antagonist in the film I, Robot from 2004. From wikipedia[0], "in [the antagonist robot's] understanding of the Three Laws, she has determined human activity will eventually cause humanity's extinction, and as the Three Laws prohibit her from letting that sort of thing happen, she rationalizes that restraining individual human behavior and sacrificing some humans will ensure humanity's survival"

0) https://en.wikipedia.org/wiki/I,_Robot_(film)#Plot

Asimov explicitly added a Zeroth Law of Robotics when he bridged the Robot and Foundation series, although it's still commented that a robot actually killing someone is too much of a cognitive dissonance to bear.

It's worth noting that most of Asimov's robot stories are him basically exploring the problems with the Three Laws; The Naked Sun is basically an entire novel showing how Three Laws robots can be capable of murder.

Interestingly, I believe that what we perceive as our intents/wants/needs/motivations is just an internal genie. Put another way, we have no idea what we really want, consciously.

Look at the Schindler's List Netflix example. Also, "A person can do what they want, but not want what they want."