National borders are not arbitrary, imaginary lines. One of the fundamental attributes of a sovereign entity is the ability to define "inside" and "outside" and control the flow between "inside" and "outside" just as that is one of the fundamental attributes of a organism.
People everywhere are not the same. Those in your country are bound together with you in the social compact. Now, it's good and necessary to allow immigration to a certain extent, but in doing so the only consideration should be what maximizes the prosperity of those who are already part of the community. Anything else undermines the social compact, and those who lobby to systematically undermine the social compact in this way to depress wages and boost corporate profits verge on sociopathy.