It happens rarely enough, and as I mostly drive on the same roads, I know where it happens as well, usually I disconnect the thing before it gets to the problematic place.
So no, it's so much stressful as annoying. But it does remind me that the claims of self-driving is just horse-shit.