Does this argument still apply on Olympics? They are nothing like average people, so I'd guess both genders push the body to the limit.
Consider the 100m dash.
In 2016 Olympics, in the final heat, the slowest man ran 4% faster than the women's world record (which has remained unbroken for 30 years).
Innate differences between the sexes are enormous, even at elite levels.
To put it another way, the difference between a near-average man and a near-average woman (per a normal approximation) could easily go either way. The difference between a top 1% man and a top 1% woman is much more likely to be skewed toward the man.