The real problem is that another country with lesser ethics will work their workers into the ground and out-compete your country in terms of productivity. Then, they'll get to effectively make all the labor rules because they'll have so much more money and can leech your workers away for higher salaries.

The real solution here is to prevent worker-leeching but that seems incredible draconian (and you'd have to get every single country to agree to it).