Accidents still happen, but it’s hard to believe they would be less safe than human drivers, when most accidents are caused by fatigue, reckless driving, illegal maneuvers, and unusual driving on the part of the operator. It’s even harder to believe that it wouldn’t become safer with effort, when both the road and the behavior of cars is far more predictable.
Most fatalities, when humans are driving, happen because of speeding (ex: 45 mph in a 25mph zone), drunk driving, fatigue/distraction, etc.
Most fatalities, with AV, will come configurations where you cannot beat the physics and stop, even with superhuman reflexes. For me, this means situations with higher speeds. The system will have less time to compute everything, and if there is an impact, the risk for fatalities is far higher: https://nacto.org/docs/usdg/relationship_between_speed_risk_...
At slower speeds, you can build a system with a defensive driving style that mostly eliminates by design the configuration where you could be the cause for a fatality. As a manufacturer of AV, you would probably want to start deploying your system where it is the least likely to fail / cause deaths.
Also, as you said (and it is probably true), most fatalities happen off highways despite the disproportionate number of miles, which means human driver are actually pretty good with highway safety, making it harder for an AV to be better than a human.