I agree that this is a problem for men. However, I don't think a clothing exchange/rental is the way to go.

If I find a pair of pants or a shirt that fits me well and I enjoy, I will keep it and wear it repeatedly until it is too worn or stretched out.

In a lot of ways, that's the opposite problem form what this site is trying to solve.