It's a category error to say that everyone else are good just because they're not the worst you can imagine. It doesn't matter what your worst 0.05% do. It doesn't make most people actually good drivers.
The nice thing about a future where all cars are self-driving (this probably will happen) is that it won't only be the taxis that get replaced.