It's important to note that it's not just bureaucrats or dictators who insist on enforcing passport and visa control. For better or worse, a lot of regular citizens of country A would be against citizens of country B staying in, or just entering country A's borders.