Gustin and Everlane.

These two clothing companies provide 100% transparency on their sourcing and labor practices.

You absolutely can choose to wear clothing that doesn't enslave people. Maybe not 100% (shoes are still difficult to source), but it doesn't mean throwing your hands up and walking away. Or being cynical toward someone who is trying.