Look at it this way: if driverless cars are widely adopted in Country A, and the new Country A government decides to go rogue and oppressive on its citizens, the rogue government could prevent people from getting to protests by shutting their cars down remotely.
This is similar to "tin hat" theories: they favor the dramatic and insidious, but unlikely, fear over the much more imminent and mundane problem.
I'm more than willing to give up the right to absolute control over my vehicle in return for better safety and faster commutes. The lives saved in both time and accidents would far outweigh the remote possibility that it would be used as an instrument of oppression. Maybe folks can control their own vehicle when off of major arteries, but be required to give up control on heavily shared routes (the same as they have to when taking public transportation.)