Also, the physical differences which matter to some sports (again, why is this in a passport, my passport doesn't say if I'm qualified to drive race cars or if I am allowed to possess* a rifle for shooting competitions) relate not simply to what's between your legs, but to the hormones which are merely correlated with that. The effect of such hormones is also why some trans people ask for puberty blockers, and in any case there's a significant black market for oestrogen and testosterone boosters.
And if you do care specifically about what's between people's legs, because you want to know how many urinals to install or something, then it should be fine to change the document (but again, why a passport?) as soon as genital surgery has been completed even in the absence of any impact of hormonal changes.
Most extreme contradiction of the "protect women's sports" talking point I've seen? As muscled-up as Dwayne Johnson or Schwarzenegger, bald, beard, no dick, identified female at birth.
* Outside the USA, we don't have or want the 2nd A, so yes a gun would need a licence.