You haven't understood my simple algorithm which does exactly what you say a human driver is doing, no advanced ml required just basic understanding of projective geometry. If you have frames of video of the previous few fractions of a second and information on how the car moved in 3d space over those frames, I'm saying you can construct a dead simple filter that preserves precisely the in scene objects which are on a collision course with the camera while everything irrelevant in the scene gets washed out.

The simplest case is with no steering. The projective transform is just blowing up the image by a multiplicative factor dependent on the distance moved per frame. So the x, y position of each pixel gets stretched the further back in time it is to compensate for the cars cha ge in parallax. After stretching, all these frames are averaged together. Anything on a collision course with the camera is going to be at the same angular position each time just getting larger. So averaging over will highlight its presence. The rear bumper which you are actually due to miss will be drifting in angular position in each compensated frame, so it gets averaged into the background blur of irrelevant objects.

Tldr simple video frame filter highlights exactly which parts of image frame relevant to possible collision, tells computer what can safely ignore, implements exactly your near miss driving algorithm, no fragile machine learning bs.