That price label may not fully represent the actual economic cost of the good (if the environmental impact externalities are not factored in). Factories could pay market price for all materials and labor, but cheap out on waste disposal. If that waste flows downstream and causes adverse health effects, the cost to that economy is not reflected in the clothing production. Market price for cotton could be the farmers who over farm their land. This price would not include the damage done and impact it might have on farming food years later.

You're right that we need to be fixing the issues at the level they appear, but every level is being squeezed for the best cost. It's up to the retailers, consumers, and fashion companies to place a larger value on environmental impact and pay the proper price.