You highlight the big challenge with computer vision and other computerized identification of things, regardless of the underlying technology. There is no "mental model" of what makes sense against which to check observations. There can be coded rules, but if something passes them, there is no extra sanity check like a person would do. Put another way, models have no common sense, and we don't really have a meaningful way to give it to them. This is going to remain a problem for self driving cars and is a big part of the reason why they have not lived up to expectations.