I think a lot of people thought it would be exactly as hard as it has turned out to be but were dismissed as being luddites or lacking imagination.
There may have been a lot of people like that, but there were very few knowledgeable people who were making such predictions in public. The only one I know of is Rodney Brooks, and even he went public for the first time only in 2017 [0].
[0] https://rodneybrooks.com/edge-cases-for-self-driving-cars/