I doubt that it's human drivers that are the issue. Cars are nice big easy to identify predictable things, and even if you screw up and hit one the cost is relatively low.
The challenging cases are probably things like children playing on the side of the road, pedestrians, bicyclists, construction workers, cops standing in the street, people working in manholes, and so on. Relatively small, easily obscured, unpredictable, hard for computers to figure out how they are moving, and very high cost to hit.