I duno, what you say sounds right but my experience seems to be the opposite, i can never find anything that fits in physical shops, everything seems to be stocked for either obese or tall people, i'm slim and on the shorter side (but not very short)... alternatively perhaps I just expect shopping to actually be efficient, i hate spending ages shopping, but perhaps frequency is the only way you can find a good fit... and the only thing I have managed to figure out so far is that fit is important.
My theory right now is simply that on average women have the will power to spend enough time and energy on the task regardless... men might have the desire to look good, but when faced with the reality of time and money required to achieve it say fuck no, it doesn't hold their interest enough.