Autonomous vehicles are dangerous in unpredictable ways. People might be injured or die in accidents that would not have happened with human drivers, as the result of software bugs rather than human decisions. When things like that happen, statistics about accident and death rates being lower with self-driving cars are beside the point.
Its politicians, who are all about "click-bait-emotionally driven" to bring them votes, you should worry about.
If you're concerned about the costs of a driver, I grant that the marginal costs per hour of driving is low, but the material costs seem high, unless a significant cost breakthrough is found in lidar production, or a technique breakthrough using multiple cameras; and the R&D costs seem pretty enormous -- people have been seriously working on this since the 80s and it's clearly closer than fusion, but it seems perenially 15 years away.