So your logic doesn't apply for your home, for your neighborhood or your city? Does it hold for you state or do you need to become completely independent country?
You need a bureaucracy once you have a shared resource, and at that point no one is entitled to the resource more than anyone else.
The land your home is built on is part of a shared resource: the space available for the community.
If we apply your logic there s nothing wrong with immigrants showing up and colonizing natives - be it in America, Australia, etc - shared resource bro!