Diversity doesn't make places more dangerous (if i understand the stats). But humans are naturally tribal and fear those who look and act significantly different.
Within a few generations, their decendants marry each other.
Unless there's a big strict enforcer to keep everyone in line of course.