Yes and no. Elite colleges have another purpose, which for many will be the primary purpose - it's how the next generation of the elite meet and get to know each other over the course of a few years, with plenty of leisure time in which to do so. Membership of the Bullingdon Club or Skull And Bones will obviously have a far, far greater effect on the outcome of the rest of your life than a generic bachelor's degree. To attract the elite therefore, you need good amenities, because that's what they're really there for. It's also why "Harvard dropout" carries very nearly as much prestige or social signal as "Harvard graduate" - getting in is the real goal, getting out with a degree is just the icing on the cake.
Similar is true of "party schools", or certain colleges where everyone just does activism full-time and no-one goes to lectures anyway.