I think any automated vehicle, no matter how advanced, should always have a big red button to make it stop whatever it's doing. Kind of like the emergency brake on a passenger train.
The difference is just that it’s higher level controls than a steering wheel.
If you hit the big red button going 65mph on the center lane of a crowded highway, it could be bad. Especially if the tech starts to feel safe enough that cars can travel close enough to use the draft from cars ahead.
I think an emergency brake works for trains because they’re isolated.
Oh wait..
A "pull over" button could work, but still relies on the automated driving capabilities.
A third option would be to dial an "expert driver" who takes remote control of the car to get you to safety, bypassing both the automated driver and the passengers' inability to drive. Lack of connectivity could be an issue, and it opens the door wider for hackers (already an issue though). However I have to imagine that in-transit connectivity will be increasing dramatically once people want to work or watch Netflix during their commute.