That doesn't stop the fact that you first have to identify that the car will not make an action automatically and by that time it is too late for a human to take manual controls back. That also doesn't stop people who are looking out the window but not really focused on what is going on since they likely spend hundreds of hours looking and nothing ever happens.

Human brains were not designed for this kind of task. Everything about the task prompts a resting mindset but the task requires 100% full concentration for hours.