I think it’s both true that most people do a lot of growing up at university and that it can make the experience worse if you are younger than everyone.
From a U.K. perspective one issue is that many student things generally assume everyone will be over 18 and therefore able to enter places that serve alcohol and only admit people over that age. Even if you don’t want to drink you may still be barred from participating. But this might not be such a big deal in the U.S.
If you’re only interested in your studies and you excel at them then maybe things don’t matter so much but I think for most people, university is mot so much about learning the subject you are reading as it is about being socialised into the middle class, and this is harder to do if you come in with one hand tied behind your back.