I tend to think of it as people having different interests. Many people have a familiarity with celebrity gossip that rivals my familiarity with programming. They're just choosing to use their brainpower in a different way.
I'm sure many people on HN disagree, but that's how I see the world. Intelligence neutral.
I choose to hang out with people who are interested in things that I am. I have a natural filter -- I continue to talk about things that I think are interesting in all company, and when conversation moves in a direction that I don't like, I disengage. That's not to say that I don't try to take an interest in things outside of my realm of experience, but people who know me learn pretty quickly that if they talk about football, I'm going to get glassy eyed. Or worse, start talking about the use of graph theory in tournament scheduling.