As another poster mentioned, "people will find just about any reason to believe they are better than other people".
Your intuition about racism/xenophobia is probably correct. But that's not the interesting question, given how widespread it is. What is more thought-provoking is why did this kind of behavior evolve? What was the advantage? Is it still advantageous today?