As someone who is often mischaracterized in a "them" category, I have the same frustrations.
For example, my dad, rather often, dismisses real conversation as me being a "liberal", a group which he refuses (possibly cannot) define. Rather than tackle the ideas themselves, he chooses to dismiss them, and, whether he recognizes this fact, uses "Us vs. Them" as a personal excuse.
Because this is an instinct, we are all susceptible. I certainly do not want to blindly consolidate individuals into an undefined group, and dismiss their ideas. I would much rather trust a person's ability to have honest perspective that is different from mine, and learn from that perspective.