What about XX with, say, Congenital Adrenal Hyperplasia? They have excess androgens and sometimes ambiguous genitalia. Estrogen insensitivity syndrome also comes to mind, where the body is unable to process estrogen. Those XX individuals have the body shape usually associated with an XY.
Or perhaps XY individuals with 5α-Reductase deficiency? They cannot process androgens and are sometimes born with genitalia that look like one typically associated with XX. They used to be mistakenly raised as individuals of the female gender. In a similar vein, there's Aromatase excess syndrome where the XY individual has male genitalia at birth and female secondary sexual characteristics at puberty. What is their gender?
Are XXY females and XXY males their own separate gender? Then what about XYY and XXYY syndrome?
Even ignoring those rarer genetic differences, what about women with polycystic ovary syndrome who are unable to conceive and have high testosterone levels? Or XY who went through menopause and have their estrogen and progesterone production limited or even halted, do they stop being women?
There's a reason both the words "gender" and "sex" exist.