You're not far off the mark. I recently read that some $28 billion worth of merchandise is returned every year due to poor fit. Though I have no idea what percentage of the total expenditure that is, judging by the effect it has on the major driving forces of fashion (namely, none), it must be pretty small. About 4-5 years ago, a company in California did conduct some research into the sizing of women's clothing, and some of that has led to changes trickling into what's being offered on the shelf. But the research was conducted only on women, and the company doing the research even admitted that their population sample was insufficient. From what I can tell, all of the basic sloper patterns that ready to wear garments are built from get their standard sizing from... nowhere. Or a mysterious table of measurements dating back to the 1940s. Take your pick.

Compound this with the total lack of quality control in garment manufacture and you begin to see why shopping is such an ordeal. The same brand and fit (model?) of pants from Dockers, Levis or the Gap will fit you differently, depending on the color of the fabric. Typically, the darker the color of the garment, the more snug the fit. Sounds ridiculous, right? Lighter fabrics tend to relax out more. It's all about the fabric content. Raise your hand if you've ever washed something and had it either get smaller or bigger, or become completely misshapen. Quality control is done on the first hundred or so garments that come out of the factory. After that, it's a free-for-all.

People don't know what suits them because there is a very real learning curve to the whole process of getting yourself dressed. This is why most women shop and dress in groups. Institutional knowledge is hugely valuable. So is a cloth tape measure.

I watch What Not To Wear. They make it look easy. Then they make it look hard (the "victim" almost always cries). Because drama sells. But behind the scenes is an entire staff of people who follow fashion trends, understand a great deal about ready to wear garments and know where to go and what it costs. That's a full time job. Nobody has that kind of time, and it's a barrier to entry.

Not caring about what kind of statement our clothes make is a learned behavior. The fashion industry does not care about the happiness of the customer. If you don't believe me, find a girl who won't slap you for it and ask her about her bra. So if they don't care, why should we?