People have this weird psychological effect around control. They will accept higher failure rates if they believe (correctly or otherwise) that they have control over a situation. Or rather, they will believe that they can beat the statistics at least for themselves, as long as they believe they have control over the situation, and achieve a lower failure rate then the driverless car. In other words, "the higher failure rate might apply to the average person, but I'm above average".
It's related to faulty risk assessment: it's much more likely that you'll get in a car and die than it is that you'll be killed in a terrorist attack, but most people don't even think of getting into a car as all that risky, and yet we start drawn-out, unwinnable wars because people are afraid of terrorism. People believe that they personally won't die in a car crash because they have control of the situation and that they're a good driver, but they have no control over when or where a terrorist attack will take place, so it's somehow a worse threat.