Only how (or if) one chooses to include those few who don't fit these categories is fluid.
In other words, there's some uncertainty at the margins, but in the vast majority of cases, biological sex is clear.
If you can't clearly define a category, any intrinsic traits of that category are not ascribed clearly to individuals. Any uncertainty, even on the margins, makes the definition uncertain. Clear definition is a lack of uncertainty.
The fallacy is the argument that two states or conditions cannot be considered distinct (or do not exist at all) because between them there exists a continuum of states.
In other words, your reasoning is a textbook example of this fallacy.