Counterpoint: personally I don't avoid doing things on the road from fear of the person behind the wheel but for fear that their reaction will cause an accident (swerving, slamming on the brakes, etc). Self driving cars have the same problem and honestly are probably less predictable.
This might be due to the fact that I don't really care much about road rage. At least where I live, the chance of someone getting out of their car and actually escalating it into a situation that matters is basically 0. When I lived in a major city, I felt the same way but for different reasons (most people in big cities Don't Have Time For That). But I can't imagine it's an isolated opinion.