The way I see it, the sex binary is fundamentally about reproduction. It's why we can use the same concept for everything from pondweed to platypuses. All across nature, male=small gametes, female=large gametes. In humans that's driven (with the potential for things to go haywire occasionally, sure, but still driven) by the XX/XY system, so that strikes me as a reasonable thing to base a definition on.
Side note re "nobody knows whether anyone is a man or a woman until they get tested" - I'd say that giving birth or fathering a child is a pretty big clue. AFAIK the only cases where that doesn't line up perfectly with genetic sex relate to mosaicism, where I'll freely admit my intuition goes completely kablooie.
My mental scenario is one of these guys trying to go into the women's restroom they are now legally obligated to use, and a whole bunch of bystanders bringing violence to the table to prevent it.
That's the desired outcome for them. They want us to either die or comply.
https://en.wikipedia.org/wiki/For_Women_Scotland_Ltd_v_The_S...
Though in practice I wouldn't be surprised if we eventually converge on "women only" and "unisex" as some sports have done. The risks posed by self-identification are very much one-way; this isn't a symmetrical situation.