Seems like it happens whenever a car in front moves out of the way. It goes full stupid-cruise-control mode and just accelerates to the stored speed. Of course there's a lot of noise in terms of things in the stationary reference frame, but still, big red firetrucks don't exactly look like lane markings.

At the very least, as a driver, I'm slightly cautious when I see a car move out of the lane in front. I feel like the solution lies in: (a) separating the signal from the noise in the set of things moving with the ground, and (b) getting the car to 'understand' why other cars change lanes. Sometimes it's so they can avoid obstacles in the road. It almost feels like cars need to have a theory of mind of other cars. I certainly drive with an intuition of driver intentions stuck in my head.