That idea is complete poison. In a debate where you're both completely uninformed, the anecdotal evidence of experience is relevant. In any discussion where reason, numbers, research are involved, the gender/race of the person making the argument is irrelevant to the argument. This current idea that only women can talk about female issues, only X about X is pre-enlightenment tribalism.
It presupposes conflict between the tribes too. If in my philosophy I believe that through reason and evidence I can understand your point of view and experiences,there's the possibility of agreement. If you really believe that I can never talk about issues affecting your tribe, or arrive at reason to an underlying truth, there's no point in talking. We might as well just fight to see whose tribe can impose power on whose.