This is a false analogy. Men tend to be 40% to 60% stronger than women and testosterone is well linked with aggressiveness.
A better comparison might be people not wanting civilized chimpanzees for drivers.
If my daughter or wife wants a female driver I am 100% ok with that.