These differences can be in factors that impact ones desire or ability to perform certain roles. Is this wrong?
This individual person, because they are a man/woman, is unfit for the role. This IS wrong.
Is it wrong to say there are differences that make women, on average, worse football players? Is that inherently bad, perhaps even blasphemous, thing to say? I don't think so.
So why can't one discuss this about other jobs? Perhaps they are wrong, Perhaps which ever group isn't worse on average. But even if they are wrong, then the way to respond is to say that their information is wrong, not that they are wrong for saying it.
We are treating the conversation itself like it is a blasphemy that shall not be uttered aloud. Instead, we should be asking 'is this information factual or not'. And we should be taking safe guards that averages aren't assigned to individuals.