Mmmm. I hear you. Sort of. Your argument though, is not very different from the "women are taking jobs that rightfully belong to men" spiel that was all the rage after WWII.
I suspect that my argument is fundamentally based on wishful thinking. That said, you can't have your cake and eat it. The 'race to the bottom' you alude to is already happening with offshoring though.
But my central argument remains ... foreigners (based on borders determined by middle aged white men a couple of centuries ago), remain the last 'minority' group that you can still discriminate against, vocally, in polite society.