I think there is a very big difference between those when personal safety is considered.

It is the same as asking if you have a preference when you get a massage or go to the doctor since those are more intimate and likely involving a lack of clothes. Something that in those situations you would feel regardless of your gender. As a male I prefer a male doctor, especially when talking about certain topics. I realize logically this is stupid and they are a professional at the end of the day, but that is my reality.

Unlike those however, there is not really a valid reason for a man to say they don't want a female driver/passenger from a comfort or safety standpoint.

We can argue all day long about in an ideal world we would not need this, but that is not our reality.