I think you've got that backwards. If there were no Neanderthal genes on the x chromosome, then only men would have Neanderthal genes.