The left fears no such thing. There are no serious anti-immigration voices on the left.
[0] now-Senate Democratic leader Charles Schumer: https://www.youtube.com/watch?v=728SW65tlU0
I'm not trying to draw a moral equivalence. But I think the barriers to immigration raised over the last century and more everywhere in the first world are something that can't be blamed on any single party or political movement.