Hacker News Reader

Comment by merpnderp | Hacker News Reader

3. The ultimate source of information about human preferences is human behavior.

Great so the robot would see climate activist flying around in private jets pumping out several thousand times the average persons output, and promptly destroy the environment by assuming our actual goal is to destroy the environment with more CO2.

View on HN

A "sufficiently advanced intelligence" would presumably understand the inaccuracy of reducing billion's of peoples' worth of preferences down to a single English sentence, the effects of incomplete information, the effects of uncertainty on decision making, the differences in time preferences and in general the effects of adding "time" into dynamic systems rather than static models, human's inabilities to explain the real reasons why they do things to themselves or anybody else, and so on and so on.

To bring it to a more human scale, an alcoholic can both sincerely desire to stop drinking alcohol and also drink themselves into a stupor every night, and a "sufficiently advanced intelligence" won't sit there spinning trying to figure out exactly which statement "The human loves alcohol" and "The human hates alcohol" is true to the total exclusion of the other. (And I mean this as merely one example dimension, of which the "global warming" problem contains hundreds or thousands of such issues at a minimum.)

It may be a valid conclusion that the climate activist values their behavior more than what they think are their goals.

I support the idea of being physically fit. My behavior belies that.

That’s not the conclusion any sufficiently advanced intelligence would draw from the facts.

That's the conclusion that a sufficiently advanced intelligence would draw from our behaviors. People act at odds to their stated and felt goals all the time, that's why "self control" is meaningful as a skill that different people have to different degrees.

The example behavior has a long term goal that a sufficiently advanced intelligence would deduce.

plenty of human behavior is directly harmful and counter to long term goals.

We are talking about a specific example here, not human behavior in general.

The conclusion I draw from people attending climate conferences and flying is that we as a population care about preserving the environment and want the freedom to travel easily. Hopefully the AI could invent for us low impact travel methods and geoengineering to keep the environment in the ideal state.

Intelligence has nothing to do with good, morality, sustainability or even living immediately into the future.

Do not mistake the two.

good point you raise: this statement is very behaviorist in leaning, and ignores human psychology, but perhaps this is to do with pragmatic reasons

Reply on news.ycombinator.com