If there is uncertainty at the margins, the category is arbitrary.
If you can't clearly define a category, any intrinsic traits of that category are not ascribed clearly to individuals. Any uncertainty, even on the margins, makes the definition uncertain. Clear definition is a lack of uncertainty.
The fallacy is the argument that two states or conditions cannot be considered distinct (or do not exist at all) because between them there exists a continuum of states.
In other words, your reasoning is a textbook example of this fallacy.