I don't believe that is true (but I am certainly outside of my expertise). As far as I understand it, sex chromosomes (specifically the Y chromosome) are responsible for genitalia differentiation, but the relevant genes of the chromosome needs to be expressed for it to happen. Whether it's the Y chromosome itself that is inactive, or genes on the other chromosomes that supress it I have no clue about.