Licensed pilot here (learned to fly in 1987). People need to look at self-driving cars in the way pilots use an auto-pilot.

Auto-pilot is most useful in two situations. 1) long cross country legs where there is not much flying to do (just maintain heading and altitude), so A/P frees the pilot up to manage other systems, enjoy the view, etc, and alleviates fatigue, 2) flying a precision instrument approach, reducing the risk that the pilot will succumb to spatial disorientation in the setup for (usually manual) landing.

With cars, auto-drive capability will be useful in reducing accidents in two modes: 1) long duration highway driving where fatigue is a big issue, 2) intervening to prevent a distracted driver from causing an accident (rear end collision etc).

I'd be perfectly happy with a car that can drive itself in cruise mode on the interstate, but requires an alert driver on local roads (with the added bonus that if I am about to slam into something, it will brake to avoid or lessen the impact).

Something for the liability crowd to consider, self driving cars won't be able to avoid every potential mishap, but they will be able to reduce their severity. A car that can automatically brake to reduce its speed by 25% just before impact, will reduce its kinetic energy by roughly half, and the potential for injury by more still.