I don't know if you've been to New York or any other places where people walk, but vectors would absolutely be intersecting on a regular basis up until a fairly short time when the pedestrian would stop. Constantly I walk toward an intersection where, if I kept going for three more seconds, I would be pasted to the street by a passing car. But I stop at the end of the sidewalk, before the road begins, so the vector changes to zero in those last three seconds. It would be super weird if cars would brake at the intersection every time this happened. Cars would be braking at every major street on every major avenue, constantly.
What's actually needed here is some notion of whether the pedestrian is paying attention and will correctly stop and not intersect the path of the car. Humans are constantly making that assessment based on sometimes very subtle cues (is the person looking at/talking on a phone, or are they paying attention, for example).