That's up to the society, hence their definitions as social constructs. For example, when you shop for dresses, do mostly men or women come up? Articles of clothing by themselves are not tied to the biology of a person's sex... meaning a vagina/penis is not required to wear a dress.
But most of society (as it is now) has deemed that woman are associated with dresses, while men are associated with suits. That image is now changing, although slowly, with trans and other non-binary genders.
An easy way to see this is to ask yourself if you think it's acceptable for men to wear dresses? If so, ask why we don't see more of that in the workplace.