We can look the squalor of factory-farmed animals right in the eye while biting into our ham sandwich and say "damn, someone should really do something about that." Or, perhaps more commonly, "I'm entitled to this, actually."
So why would we budge over a little forced labor on the other side of the world? Or pollution? Or anything else?
We care about things right up to the point where we have to lift a finger. Though, aside, for this reason I think legislation is one of the only solutions short of waiting for a cultural awakening that probably won't ever come.