What if larger sizes of clothes were priced higher, since they use more material? I wear a small in almost every case so wouldn't affect me, but man it'd be nerve wracking for a lot of Americans.
Unusually large or small sizes can end up more expensive (and/or only manufactured in limited quantities) because they're not commonly bought and they take up space on the shop floor and in inventory which could be used for things with higher turnover. (Edit: Also at the extreme ends of sizing simply enlarging or shrinking the pattern won't work well, you have to redraft it so it sits correctly on a petite or plus-sized frame).