I used to think this, but I don't any more. One argument that swayed me came from Izabella Laba [1], a math professor at UBC and former IMO competitor. The point that stood out to me is that women tend to be socialized away from extremes, for example it is generally more acceptable (to parents, guardians, and affiliated people choosing for children) for gifted boys to skip many grades than for gifted girls to do the same. Along the same lines, obsession and isolation in boys just seems to bother us less than the same thing in girls.
There's no great comcrete evidence backing this up (as far as I know). But Laba is one who should know, having once been a very mathematically gifted girl.
Maybe men really are outliers on a greater scale, but IMO we do enough tamping-down of female outliers that I'm not willing to draw that conclusion from the preponderance of male geniuses we see today.
[1] https://ilaba.wordpress.com/2017/06/24/gifted-while-female/