Comment by ds9 | Hacker News Reader

Comment by ds9 | Hacker News Reader

Parent Full thread

Another case worse than partially-hijacked trips was mentioned in a slashdot post, and if I recall correctly, there was a vague reference to the autonomous-car-makers considering it, and it has bothered me ever since.

Imagine a situation where a crash appears to the AI of car C1 to be be inevitable. But also, if C1 executes maneuver M1, its own occupants will be saved from harm, but another car C2 will have a worse crash or will crash when it otherwise would not - but if C1 does maneuver M2 instead, its own occupants will be killed or suffer worse harm, but the occupants of C2 will be better off. What if, in such a scenario, C1 would decide to "sacrifice" its own occupants to minimize the total harm (e.g., C2 has more occupants)?

Self-driving cars might well be designed this way. It might be mandated by law. Authorities would try to reassure objectors by saying the "avoidably killed by your car" outcome is very unlikely and that you're better off in the big picture of all risks.

Edit: On a closer reading I note that the linked page http://www.theatlantic.com/technology/archive/2013/10/the-et... covers, not exactly the above, but similar scenarios.

Apparently in the UK almost every inevitable crash will end with the driver swerving right, usually putting his passenger in harm (driver sits on the right). I'm told that's a natural and unavoidable reaction, so if it's true I assume the opposite holds for America and etc. I guess that's another decision the car has to make, do you hit an unavoidable obstacle head on or swerve, presumably drastically changing the survival chances of each passenger. The baseline human panic reaction is almost certainly easy to 'beat' but it's a horrifying though that a car would have to make that decision.

I've always thought that the protection part was a fallacy and the more obvious conclusion is that they swerve in the direction they are used to safely passing with.

It is likely that an AI sophisticated enough to make the consideration will be sophisticated enough to avoid the question.

That is, there isn't all that much reason for the cars to be operating on the precipice of disaster.

(If you want to say "But what about uncontrollable and unforeseen circumstances?", we should hope that we make the test out to be 'better than most humans', not 'perfect')

Assuming we change slowly to self-driving cars and they're operating for a while in an environment with human driven vehicles, no disaster scenario should be assumed out. They will have to deal with the situation that kill people right now, and them choices could easily reduce to "kill my passengers or those drunks in that car hurtling toward me" for example.

Better than humans is great, sophisticated AI is great, we can't expect perfect but I don't see a situation where they enter an environment much less dangerous than today's.

I expect the standard heuristic when the program detects an unavoidable collision will be to minimize impact energy, whatever that means.

(It won't have good information about the mass of oncoming traffic, but there aren't that many situations where choosing to collide with the oncoming vehicle makes any sense at all)

As with most thought experiments, it's a little far-fetched though. How often, in the run-up to a crash, is it really possible to make oneself safer (or less safe) at the expense (or benefit) of another car?

A more common scenario might be driverless cars that don't communicate with each other and therefore try to "optimize" a crash without knowing how other cars are going to respond – potentially far more dangerous than the magnanimous AI you talk about.

And imagine if the AI also rated the car occupants on their future value to society.

If we're talking about Google, I assume it would rank people based on the number of friends they have in Google+.

The cynical side of me suspects that the "Volvos" of the future will make this choice less often than the "Pontiacs". The rich will still have safer cars, AI style.

It seems like an interesting branch of "applied theology". Does one, in the end, have faith that these tiny entities imbued with lesser-god-like driving abilities really have one's best interests at heart?

You appear to be claiming this is a bad thing, when (assuming it worked as planned) this would make you safer, since on average you'll be in the other car.

Of course, an irrational fear of safer cars is already considered to be one of the big stumbling blocks for self-driving cars, even without this scenario.

http://en.wikipedia.org/wiki/Trolley_problem

I need to start carpooling more

Reply on news.ycombinator.com