There's everything from $13 ones at Walmart to $70 - $90 ones at J. Crew and Brooks Brothers (MTailor price point) to $300 Armani shirts (and above).
Having spent a lot of time in in-person custom shirt shops, our $69 shirts are comparable in quality to shirts you would buy there for $125 and above.
And obviously we'd go even lower if we could.
If you cannot go lower, I'd consider a partnership with an existing brand. You'll have to have an insane marketing budget to compete with existing clothing brands without viral.
If you can partner with Brooks Brothers (established in 1818, and part of American preppy folklore), J. Crew, Men's Warehouse, or similar, you'll have an easy time breaking in, although a much harder time getting most of the profit share.
Word-of-mouth can work well, but you've got to understand your customers, the networks, and the levers you can tweak to change rate-of-growth. I was suggesting price/quality/margin as one of those levers -- you want to be appealing to as many customers within your customers' networks -- but there may be others if that one is unappealing.
Compare the market caps of Walmart, J. Crew (which was recently purchased), Brooks Brothers (from a sane P/E ratio), Armani. Heck, try a TJ Max or similar (to avoid companies with major non-clothing businesses -- which, coincidentally, J. Crew does have).