There's a saying in software, you throw away the first version. The main idea is that when you start on something, you don't know what you don't know. The labor of the first version allows you to learn the domain, the limits, edge cases, and what you need to do. Then the second version starts on a much better foundation.

At this point, many teams (Waymo, Tesla, Uber, etc.) have been working on this for years. And there must have been times when the engineers thought, this would work so much better if only... Maybe it's time to gather these thoughts and sort them by low effort/high reward. For example, if all cars had a cheap sticker indicating forward vs back, would that improve self-driving performance by a lot? It wouldn't be that hard to get car manufacturers to add a sticker somewhere. Or what about traffic lights? Maybe adding a cheap light or sensor to it would vastly improve self-driving performance. Or those road reflectors, can they get some minor changes to help the self-driving software?

It doesn't seem likely that level 5 or even level 4 will happen when self-driving is literally defined as replacing what a human driver would do. But what if the computer had some help from both infrastructure and other vehicles? Why not start thinking about cheap changes that can be implemented to fix the blind spots and flaws in the self-driving software? If these changes are cheap and easy to implement and would vastly improve the software performance, isn't that worth it for true self-driving to work?