I would take the hacker culture of Google or Apple over drunk brosefs driving from one bar to another on a Saturday night every time.
In fact I hope for the day that driving on public roads would become illegal by humans.
My big worry is that we'll have autonomous-car-AI monoculture. If that happens, we'd lose the advantage of the "compensation" other drivers in traffic can do. We could end up seeing car-behavioral "prions": glitches in car software that cause a strange behavior, which in turn triggers the same glitch in the cars attempting to compensate for that behavior. You could see, for example, every car getting into a state where it thinks it needs to swerve "more right than other cars" to avoid an emergency, and thus an entire "pack" of cars driving off the road and into a fast clockwise-torquing circle.
Or, less eyecatching but more fatal in the near-term, you could see a monoculture-wide decision that works alright for single driverless cars, but actively harms drivered cars (and other driverless cars with other-model AIs) when the majority of a "pack" of cars is from the monoculture. For example, I could see a "pack" of driverless cars all deciding to jam on the brakes at once (because, say, a bunch of spike-strips suddenly popped out of the road in between them or somesuch)—leaving the one drivered car boxed in by them to keep going and crash into the vanguard of the pack.
Sure there will be some accidents, but nowhere near the same frequency as today. Autonomous cars are much better/faster at "compensating" than humans are.
A by-product of this might be an airline level of safety and after the accident investigation. There certainly would be the financial benefit for it. Once the autonomous car comes it will be safer than humans to start with and only become safer and safer with time. It might get to the point where we regularly travel at 90+ mph on the highway with barely a care in the world.