When you give a man a female dose of hormones or a woman a male dose of hormones, it has a very big effect on their mood, behavior, and mental wellbeing. This change is much, much bigger than the average diffences we see between men and women. For example, an average man with an average woman's level of testosterone will experience a MUCH higher level depression, listlessness, and sexual disinterest than the average woman experiences.
This strongly implies that human brains must correct for these huge hormonal differences. Basically, in order for male and female behavior to be similar, their brains must differ. If their brains are the same, then hormones will have a much, much bigger influence on male and female behavior than what we actually see in reality.
Hormone-correcting brain differences would also imply that it's possible for people to be born with some type of intersex brain condition, and that these individuals would benefit greatly from receiving hormone therapy to bring their hormone levels in line with their brains. And this, indeed, seems to be something we see occasionally.
(In case anyone cares or thinks it is relevant, I wish to note that I am a cisgender woman and I do not think that there are huge innate differences in men's and women's mentality — certainly nothing like on the level that testosterone/estrogen/etc levels would predict. I think most of the differences we do see are environmental, which is why these gaps have been closing in recent history — or widening in some cases/locations. Based on these trajectories, I suspect that men and women are actually FAR more similar than anyone natively groks, and that we exaggerate or invent small differences due to a biological hyperfixation with sex. Note that we don't obsess about the mental differences between, say, male and female cats, even though they have much greater sexual dimorphism than humans do.)