A colleague of my dad's tripped and fell while walking down a set of three stairs, hit his head, and died. It was nobody's fault -- it was just bad luck.
If an airplane autopilot sends a plane full of people into a mountain due to a defect in the software, who is liable?
Even on autopilot, I'm supposed to be monitoring the systems, keeping track of where we are, where we're headed, the weather, etc. If my autopilot sends me towards a mountain, it's on me (practically and legally) to fix it.
Edit: Actually, this raises a question in my mind. Do autonomous military robots have "handlers" or the like?
The decision would need to be made case by case in courts to determine who (if anyone) is responsible. At least in court systems like the US has. For very different systems, I'm not sure.
Can it prevent all of them? No, but mine has a mandatory self-test that has to complete before it can be engaged, there is a dedicated breaker with a collar so I can kill power to just the autopilot, failing that I can kill the power to all avionics, failing that kill all electrical power, and failing all that each of the servos has a clutch on it that can easily be over-ridden by even the daintiest pilot. If the autopilot is commanding a left turn that I don't want, I can mechanically override it, or I have 4 electrical ways to shut it off. That takes "autopilot can't be disengaged" way, way down on the list of things to worry about, especially when compared to flying behind a single piston engine, with no anti-icing equipment.
If I fly my airplane into a mountain, the NTSB report will say: Probable cause: The pilot in command's failure to maintain clearance from terrain, resulting in in-flight collision with mountainous terrain. They may add: Contributing causes were: <some list of lesser causes such as weather, lighting conditions, failure of navigation equipment, projected spatial disorientation, and if they have evidence to support it, autopilot malfunction or more likely, pilot's incorrect or insufficient reaction to that malfunction>. The likelihood that S-tec [maker of my autopilot] is found liable is precisely 0.00%.
I suppose there is no need for me to point this out, but I'm not a pilot and have never flown anything, with or without autopilot. I was just using this as an example of "something going wrong with the machine" to show that there are times you would blame the pilot and times you would blame the company who built the autopilot. If autopilot is a bad example, replace it with some other autonomous machine... or pretend that we were back in the stone ages of autopilots when things could go wrong. Or, I'll make up a new example where the autopilot all of a sudden inverts the plane and then sets it right before you get a chance to turn it off therefore injuring one of the passengers. Lets not get too caught up in the specifics of autopilots and aviation laws, etc...
That said, having a number of failsafe mechanisms (like the ones you described) would be a critical part of any autonomous system, especially in the early stages, which would, of course, affect who we end up blaming.
I wasn't trying to argue against any particular point of yours.