From what I have seen: true. (Dartmouth alum here.) Much of the benefit of going to an Ivy League school is the chance to socialize with and develop interpersonal relationships with "other upper-custers." (Not something I did.)
(networking with smart people is of perhaps maximal importance).
Even the kids at the private high school that were on scholarship and came from poor families were not the same as the kids from poor families at the public high school (once again, as a group individual kids of course could vary).
Add: I came from a middle class family.
Me too.
> exposure to people from families that are tremendously committed to education and were raised with certain values.
That's to be had. You can also meet lots of cynical elitists who see themselves as superior human beings and exempt from certain moral codes regarding the treatment of their perceived inferiors. Of course, this is true at just about any school.