I firmly believe that the literature produced which shows self-driving cars are safe or even safer, and there is scads of it now, is mainly down to the program's ability to make sure a safe following distance is observed leading to greater reaction time for braking. I know a lot of driving tests have been done in simulators which show the AI can be safer but I agree with your statement.

I also have very little confidence in the AI's ability to stop at road construction, make its way over temporary embankments and diversions placed for construction, navigating in inclement weather or adverse traction situations like a muddy drive, etc. I suppose the real goal here is for Wal-Mart and Amazon to move freight on the Interstates to begin with but I don't even see that happening any time soon. Your average Wal-Mart loading bay and parking areas are likely much more than any current AI can handle anyhow. How does the AI know the person in the mart cart isn't going to just jut out in front at the last second? Can't exactly make eye contact.