It's not that going to some particular school or even a set of them (Ivy or no) matters per se. The effects of privilege exist on a continuum, and tend to accumulate over time. In that article I linked, the filmmaker had fairly extraordinary privilege via a long accumulation of access, education, peer mindset, parental support, and resources. But that's perhaps an extreme example. A lesser example: just the fact that a university student was even able to consider college as a life path is also a manifestation of privilege. For the most part, it points to an accumulation of parental and peer support over the student's life. E.g. her parents valued school and encouraged schoolwork, maybe tried to get her into a better school. She had peer support for life-paths including notions of education, career, "bettering oneself", etc. Peer support is especially interesting, as it also subsumes a lot of class issues.
Memes matter, and speaking probabilistically, kids whose peers see no paths to "success" in life can expect to have a hard time finding it themselves. We've likely all seen tales of hugely successful people who rose from modest backgrounds. That's the old story. The new one, the one we're just starting to tell, is about the enormous waste of potential from all those who didn't make it.
There was a great cartoon that the farnam street brain food blog sent out which illustrates this quite well.
http://thewireless.co.nz/articles/the-pencilsword-on-a-plate