That may be true logically, but I don't think it is workable politically.
People are conditioned to accept that we will kill each other with cars every once and a while. I'm skeptical we will get to the point any time soon where the general public hears about a family killed by a self driving car and just shrugs it off. There will be intense pressure to get them off the road.