I think you might be confusing legal and illegal (or documented/undocumented if you think “illegal” is too on the nose) immigration.

You can be anti-illegal-immigration (which is effectively being anti open borders) without being a xenophobe that is wholly “anti-immigration”.