I'm a 5'5", 110lbs man. I shop at the teens section and get larges. I may not get the trendiest looks, but I get cheaper clothes that fits and looks good on me!
I also tried Stitch Fix, they had a surprising amount of stuff that could fit me (both fashionably and size wise), albeit not as cheap as kids' clothes.