What makes people who don't listen to the music associate it with 'negativity' is not just the music itself though. It's the way some fans dress as well. Or the mosh pits / wall of death etc they might hear about.
People are quick to judge, and the way you look will factor into that.
But it's a generalisation people make. It happens that people whom I've known for some time react surprised when they find out my favourite genre of music is metal because I don't fit their 'picture' of what someone should look like who likes that kind of music. :) (At least, if they haven't seen me at home with band shirts on)