The segregation is not all systemic. I remember reading an article discussing research discussing how neighborhoods change as a result of very slight preferences of "I'd rather be by somebody of my race than a different race." This applied to black and white neighborhoods.

Taken over time and populations, these very slight differences in preference (and, realistically, in the USA, it is unlikely to be "slight") lead to segregated neighborhoods.

If anybody has the original, I'd love to see it again and find out what I've mis-remembered. I couldn't find it.