Me. But with no authority. My individual opportunities are far greater if I'm part of a society that ensures equal opportunity for all. Much like how having a large population of immunized individuals far outweighs an individuals rights to not receive immunizations, living in a population where no one is allowed to attend an inferior school improves my own life opportunity. Educated people live longer, healthier and more peaceful and productive lives -- I would personally benefit in a scenario where everyone in my community was educated fairly.
There is no way that poor people demand to live in slums. The self-segragation idea is insulting and couldn't be supported by even an uninformed hypothesis.