The biggest hurdle appears to be human error. 'What about drunk drivers who drive on to your side of the road etc'. My question is, what if all cars are automated? Zero drunk drivers, minimal risk of encountering unpredictable driving behaviour, much easier and safer to implement.
It will get to a point, during our lifetime, where humans controlling a car are limited to private race tracks. Speeding, drink drivers, poor driving ability, insurance premiums, all of these will push automation to the forefront in no time at all.