I guess because the rivalry would have no meaning and no real benefits, and could lead to needless aggression. And because having conflicts with faraway people I'll never meet is bad enough; having conflicts with my neighbors is an unsettling proposition.
How permanent is that solution? If a child born on the opposite side of the street had access to good education, health care, and jobs, could they also become "intelligent, wealthy and altruistic," and if so, wouldn't it be in my best interests to help that happen? Or even for no direct benefit, wouldn't I just want to help because of that altruism you mentioned?
Conversely, if they are kept isolated from us, wouldn't it make it even harder for them to achieve that potential? If so, doesn't their enduring poverty and lack of intelligence and empathy mean that they're more likely to grow up to commit crimes on my side of the street?