well...yes? For example: wouldn't you be more upset if a collision was caused by a drunk driver than by a driver who was paying attention? That's because drunk drivers are known to be dangerous and should not be operating.
If self-driving vehicles overall increase safety, then I am not as upset as I am if they are known to overall decrease safety, because the fact of their presence being just or unjust is pertinent to the case.