In my experience it is those who love to flout their sense of superiority by shitting on users who are arrogant. But shit can flow both ways and when I'd point out that they too use "drugs" they'd get very defensive.
Back in the 80s my cocaine using friends loved shitting on weed users. "It makes you dumb" was one of their favorite cut downs.
My alcohol drinking friends who didn't smoke weed loved to cast the aspersion that weed smokers were "drug users". When I started pointing out that alcohol was a drug too they all very loudly denied it because couldn't stand the idea that they were "drug users" too.
And along with and within those two groups are those who do prescription drugs. It was pretty strange to be accused of being a "drug user" by people taking codeine and Xanax and Quaaludes, ect.
When I pointed out the serious side effects of alcohol, both personal and to our society at large as compared to weed smokers they started shutting up about that when talking to me.
Since Colorado legalized it those kinds of encounters have been far less common, and I've certainly appreciated it. And because of that I'd probably be far less likely to be accused of being arrogant nowadays.