The problem is that you could apply that argument for a drunk driver, a distressed driver, a driver without a license, a driver that hasn't slept enough, etc.

Any of those could mean a car could randomly turn into the crosswalk.

Humans are horrible at assessing risk. We worry about Teslas because it's on the headlines now, but you're infinitely more likely to get ran over by one of the cited examples than a Tesla just by sheer statistics.