Well, the implicit connotation in the term 'privilege' is that it's been conferred as some sort of favour, perhaps even a temporary one (e.g. it's a privilege to be awarded a prize).

Compare with, say, 'advantage'.

Underprivileged, and disadvantaged, are also often used as synonyms within the social context, whilst outside of that they're not at all the same thing.