I learned to contra dance in a groovy, gender-free group, and I found it frustrating for several reasons.
Contra dance is designed for bodies that fit together, statistically, as men's and women's bodies do, and it was far more often the case that a figure designed for a man and a woman would become physically awkward when attempted with two men.
Many of the figures involved four or eight partners, and it's honestly confusing for beginners when you lack a visual reference point for whom you're supposed to dance with next.
Dance is a pretty mystical thing, and I don't find the question "why should the sex of your dance partner matter?" to be that compelling. It matters because I want to dance with women who want to dance with men. I don't begrudge anyone for setting whatever ground rules suit them, but I wish the STRICTLY GENDERFUL, PROBLEMATIC AND PATRIARCHAL dance groups would advertise themselves a little more loudly.