How many colleges provide access to the upper crust of society? 10? 20? The vast majority of kids aren’t getting into them anyway, so it’s a moot point for them.
Depends on what "upper crust" means to you. Political class? Perhaps not. But it still offers some peer selection that's important for life outcomes for most people (e.g. lower chance of divorce).