If the model statistically significantly returns a different number for men and for women when controlling for all other factors, then it's a bias. I don't understand how this is even contentious.
The weasel phrase is "controlling for all other factors", which is actually impossible.