Why shouldn't surrounding infrastructure adapt a bit for self-driving cars? Look at everything that had to change for railroads, cars, and airplanes.
If railroads were invented today they would never be allowed. They can't stop for miles? You have to teach everyone everywhere to stay off the tracks? How's that supposed to work?
It's good that our safety standards are a lot higher today for new tech, but perhaps there are a few common sense rules that could be taught, rather than requiring perfection?