If other people steal the emblems of my identity (by liking the same band I like, or going to the same conference I do), it's as if they're stealing my soul. Worst case scenario, I might have to admit that I'm not that different from the average guy. That's a bitter pill for a lot of us, especially those who were raised with or otherwise acquired the idea that we're exceptional or superior.
An interesting irony about this is that the ego dynamic underlying it all is so primitive and so identical.