> but on the other hand I really do wonder what the limits of the current approach is.

If the technology stagnates, the hardware stops improving, and the software stops getting better, then we would hit the limit pretty quickly.

> After all humans don't drive by analysing 1 frame at a time, we have a model of how the world works, we anticipate, we have intuition etc...

All self driving cars also maintain those models as well, and use frame by frame input to correct those models in real time (but they have already guessed where something is going to be 100+ frames from now).