From a marketing standpoint, I wonder how you actually find the customers you're looking for. Most start-ups seem to generate enough interest through HN, TC, and other niche focused sites. However, I'm not sure these are your key audience.

From my own shopping experience, I generally find one brand of shirt/pant/shoe that work well and then it's my go-to brand. Anytime I need to buy something new, that's where I go. I hardly deviate or try something new because as you mentioned, I hate shopping and as other people have mentioned, fit and feel are important.

However, I think you can expand your product by saying "hey, if you like Shirt A, then maybe you'll like Shirt B". After all, the style from one brand often remains the same and even guys will want different types of clothes for different events/moods/etc.