Hacker News Reader

Comment by 29athrowaway | Hacker News Reader

AIs are the best at exploting unexpected perverse incentives.

You will see this whenever you train an AI to play a video game. It will eventually find a glitch and exploit it endlessly in the weirdest of ways.

If you poorly design an AI to lower crime rate in a given population, it can come up with a solution where population = 0 and therefore crime rate is 0.

View on HN

This poorly designed AI you speak of is the premise of I, Robot by Asimov. In fact, this is one of the fears surrounding AGI - the AGI would be correctly configured to act only for the benefit of humanity, however, in typical AI fashion, this leads to unexpected (and often undesirable) effects.

> If you poorly design an AI to lower crime rate in a given population, it can come up with a solution where population = 0 and therefore crime rate is 0.

Since crime rates are in offenses per capita, the crime rate would actually be undefined in that case.

The mistake would setting a fixed population denominator, not accounting for deaths and demise. This would be a disconnect that would allow the rate to be defined.

> If you poorly design an AI to lower crime rate in a given population, it can come up with a solution where population = 0 and therefore crime rate is 0.

Isn’t that kinda the plot to the movie iRobot? And/Or terminator?

Another famous variation is the paperclip maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...

And what happens with the computer in "Space Odessey - 2001"

Reply on news.ycombinator.com