Well, by that logic, we should abandon all labels altogether. Down with identification of groups of people! Cast off the oppressive shackles of classification!

Down here in Australia, "Hipsters" are an easily recognizable and quite specific cultural subgroup: Thick-rim glasses, no actual vision impediment to justify them, Impractically skinny jeans, Borderline offensively cliched facial hair (ie. Hercule Poirot-esque moustaches) and - most importantly - Irrational dislike of anything "mainstream" or "commercial" (with little/no other justification)

It's no longer meaningful counter-culture, it's become a very specific subculture in and of itself (at least down here) - a culture I would describe not as counter-culture, but anti-culture. It's ceased to be productive and meaningful, and simply become (like Monty Python's Argument Clinic) the automatic gainsay of anything "mainstream".

I've no tolerance for it, because (in my opinion, at least) it's highly pretentious and offers very little in the way of positive change. I find that subculture to be, in general, very negative (often for very little reason).

When I talk about Hipsters, I'm generally referring not to the dress, but to the attitude. Maybe you're right, and I should be referring to those specific attitudes as what I take issue with - but sometimes convenient (if inaccurate) labels make communication faster.