I think the ways we are using the phrase 'economically efficient' are different, and that's where much of the disagreement lies. In fact I'd reject the notion of the existence of a universal objective welfare function. If you include the welfare of slaves in your calculation certainly I'd agree that the system is inefficient; but since when has any slave owner done that?
It's actually an extremely uncommon (nonexistent) left wing view that welfare is maximized with some people enslaved. Literally nobody believes that and I certainly hope for your sake that you don't believe anyone believes that and are just trying to be incendiary. It is a very common view among most people (since it's the truth) that the welfare of /certain people/ is maximized with /certain other people/ enslaved. And then we can conclude if the first group has economic power and is subject to no regulation and act according to maximizing their economic welfare, they will implement it.
The contention is that your position, that businesses should refuse to operate in countries which regulate rights into existence since these rights have costs which hurt the bottom line of the business, if adopted leads directly and immediately to slavery. Slavery is unquestionably disgusting and reprehensible, so therefore your position, which leads to an unacceptable result, cannot be accepted.
So I'm suggesting that the /subjective/ welfare of people in a position to own slaves (wealthy business owners) is maximized under slavery. And from that I conclude the maximization of the welfare of wealthy business owners -- the natural result of unfettered capitalism -- is something to be categorically rejected. So we need fetters.