The reason elite colleges are elite is because they are filled with people deemed "the elite." Got that?
A big part of the experience at ANY undergraduate school is WHO you happen to be learning WITH not just who you are learning FROM. Harvard made a very deliberate decision to cultivate a class composed of the BEST students from across the nation. It's easy do educate students that are ALREADY motivated, good students. BTW, Harvard also took great pains to make sure that there weren't too many Jews or Asians in the freshman class. (So I guess that signalling thing works in some unanticipated ways too.)
Just keep in mind that signals come in many forms.