Some proponents of self driving cars might have gotten too ahead of the hype train but nevertheless, self-driving is here and soon would be common place.
It would obviously take some time to turn over the current inventory of non-self-driving cars but even that would happen faster than most people anticipate. Americans already replace their cars quite often and it is hard to think of a more compelling nudge to replace your car than the prospect of not having to actively drive it.
As for the outstanding technical challenges, I am often baffled when people point to that as a reason for skepticism...it is the whole point of the endeavor, to solve those problems.