I wonder if real life will play out the same way : will controlling a car the human way become an unacceptable risk, and how fast will this happen ? Because we're likely to see both effects play here : first, humans will become worse at controlling vehicles, and vehicles will become better at it, likely having superior senses and predictive ability. So there should come a point where humans become a risk.
So there will come a point where we get to choose between new policies and letting humans drive. If we had self-driving cars and we could eliminate speed limits on interstates entirely, in trade for not allowing human drivers at all, would we do it ?