I think being sober around drunk people is like going to a naked beach with your clothes on. It just doesn't feel right, and if you don't want to drug yourself up then you shouldn't want to be around people who are.
Besides, most people are multi-dimensional, even if you don't fit in on this thing, you might fit in on another axis. Why are you even with the group to begin with? Focusing on that will be more important for social bonding than focusing on joining or not in alcoholic drinks.