Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction

That approach is fine for transportation, but not for nukes, and AI enhances robotics is closer to nukes, in many ways, even though robotics in general runs the gamut all the way down from replicants and Skynet to toaster ovens and juiceros.

It's okay if some toaster ovens burn down a few houses, and then get recalled due to a defective design, right?

But if you have a massive flotilla of self driving cars that go ape shit, flipping over, slamming themselves into trees and catching fire, and suddenly kill 10 million people overnight, around the world, due to an automatic hot fix maliciously kspliced into running kernels by an advanced actor, targeting systems are in motion in particular, that's more than a little terrible. Note that the actor could be a synthetic entity or not.

So, if you think about trouble the with autonomous systems being turing complete, engaging in tasks of arbitrary complexity, it doesn't seem that we can expect a bottom to the worst pitfall imaginable.

AI is unlike aviation, in the sense that we can't trust a simple axiom like: "what goes up, must come down."

We need to be pessimistic. Sometimes fear is healthy. Caveman logic sometimes am good. Fire hot!

You might also all "how much control?". Isn't one of the things about autonomous operation that systems are specifically designed to function in times when humans might fail.

Was the ABS operating? Oh, so the human wasn't in complete control. How about collision avoidance, did the car steer or de-throttle? Minimum levels of automation are probably did wipers operate, did headlights dip.

If the human was in complete control then that probably indicates some system failures that may have contributed.

Looks like it was posted in 2016 and updated in 2019.

Yes — and organizations have been firing (and historically enough, killing) middle managers to “crumple” diffused executive misconduct for a long time.

But “moral crumple zone” sounds sexy, while “the human in an AI driven system is just a scapegoat” is probably something the average person on the street has mused about, extrapolating from purely human behavior or in the context of “taking over” for self-driving cars.

His newsletter is A+: https://desert.glass

At the top of the page, orange button. Took me a minute to spot it too.