It's not that people will judge you IME, but that it's really fucking boring.
Unless being drunk is your idea of fun (in which case it seems doubtful you'd be a non-drinker), I cannot understand how drinking liquid A at a party is any more boring than liquid B.
In fact a graph of my enjoyment as a non-drinker (in my experience) would probably be inverse to a graph of the enjoyment of the people getting progressively more shit-faced.