In my own opinion, being a passenger in a self-driving vehicle wouldn't be riskier than being a passenger in a normal vehicle. On the contrary, I think it's safer. But there's the issue of how human drivers might behave or be forced to behave once self-driving cars become widespread.
What if human drivers' become more trusting of other cars' behavior and start paying less attention? Human drivers might become involved in worse crashes between themselves.
What if there's an adverse selection effect? If the public is convinced that self-driving cars are safer for individual passengers, then the most risk-averse drivers might be more likely to join in, leaving less risk-averse (and prone to accident) drivers on the road.
How would human drivers interact with platoons? What if a human driver is stuck in a suddenly decelerating platoon? Would he panic and swerve? Would he crash?
My point is that there might be all kinds of unforeseen consequences. Maybe good, maybe bad. That's reason why I'm not particularly enthusiastic about self-driving cars (or anything else for that matter): we'll only know whether it's good or bad in hindsight.