This is exactly my feeling.
I didn't go to an Ivy League school. But it doesn't bother me at all that children of rich people go to elite schools, get degrees in Russian Literature, Art History, Political Science, play lacrosse, get jobs in white-shoe law firms, sit on boards, etc.
It doesn't affect me or hurt me at all. Let them have their club. So what?