I wonder if this is a good application for recently developed multimodal/vision-language models that demonstrate "emergent" properties like an ability to classify objects unseen in their training data.
That would definitely help. Honestly though I feel like level 5 self driving requires a theory of mind, and I'm not convinced that we're there yet.
I keep coming back to things like you have a pedestrian on a street corner. What are they going to do? Are they just standing there or are they going to cross? Are they paying attention to their surroundings or not?
Yeah basically. We need a decent world model to handle the edge cases, and current ML approaches are 100% not there yet.