I don’t think it’s merely being around other young people. There are classes on topics that you will never ever come across in a job context, yet will have a major impact on your personality, mind, etc.
I definitely would have felt differently about college if money was not a concern. I got accepted to CMU but didn’t go because I didn’t want the debt, and at the state college I attended, I didn’t major in what I wanted to do. Looking back, I do feel some regret, but also, if I had chosen differently, I would still be paying for it today. Even if you’re looking for an amazing learning experience, the unbelievable cost forces it to be a primarily financial endeavor, at which point it makes more sense to forgo niceties like philosophy and study it on your own time, or audit classes or something.
But I don’t like it when people then rationalize that decision into “it actually sucks anyway,” because IME that’s not true.
In my personal experience, I studied philosophy in college, and have continued reading books off-and-on since.
There really is no book replacement for being in a class of 5-10 philosophy students, arguing about Kant or metaphysics for 3 hours, or in having a professor rip your paper apart and highlight every flaw in an argument. It just won’t happen if you’re self-studying.