At least part of the reason that individuals traveled freely is that only rich individuals traveled.

Paupers from London didn't get on ships to the new world - not without selling themselves into indentured servitude.

When the economic barriers to migration fell to the point that subsistence farmers could travel between continents, we were always going to see either an equalization of living standards or the rise of political barriers.