This is a bit of a generous assumption. It very well could turn out that self-driving cars are worse, or no better, than human drivers.
Not to mention that the failure modes of self-driving cars are likely to be very different from failure modes of humans, making it more difficult for other road users to predict what they would do in any given situation.
The only question is can we make them work in situations that are hard.
In other words, "it works until it doesn't" is tautologically true, but useless precisely for that reason.
On the other hand the self-driving cars are just getting started. It's entirely reasonable to suppose that they may surpass the average human in the next X years. And they won't get old and confused and eventually replaced with new cars that have to learn from scratch -- they should get more or less monotonically better. I think the optimism is warranted even if the timeline and technology is uncertain.
This is trivially true. I mean, self driving cars that are worse than human drivers already exist. Isn’t the point asking when they will be made widely available? I assume that is only possible when they are at least as good as human drivers by some key metrics.
However, I've noticed that many work from the premise that self-driving cars are already safer than human drivers. That, or they'll work from the premise that self-driving cars will definitely be safer than human drivers within a reasonable timespan.
Worse, some assume that self-driving cars will dramatically, or nearly eliminate automobile accidents/deaths, where any brief skim over workplace casualties involving autonomous machinery would put that fantasy to rest.
For a group of tech workers that often overlap with self-described skeptics, it's an interesting blindspot to have.