But this is exactly why I think that the consumer version of the self-driving car will always have a manual drive setting that will completely disengage the auto-pilot.
The insurance companies will never insure at an acceptable cost point a car that cannot yield control to its passenger that is the designated driver.
Also, passenger will want the reassurance that at any point they can take over the car and switch it off the auto pilot.
iPhones and any other mobile devices are a bad analogy with a self-driving car. The more apt comparison is with the current modern airplanes: even if these planes can drive themselves, the pilot is able at any time to take over.
This option is needed even more on the roads, where you don't have the dedicated flight path a plane has in the air.