Tenderloin is much scarier. My friend got punched in the face by a random guy walking down the street.
The worst aspect of the Haight is the gauntlet of gutter punks between Cole and Clayton. It's actually a very nice, dare I say, "gentrified" area, aside from that particular stretch.
The drugs don't bother me; it's the people sitting on the curbs, pissing on the sidewalks, accosting strangers for change, offering drugs, asking for drugs, hiding in the bushes, howling at the moon that bother me.
Ironically, I've never had a problem in the Tenderloin. I lived in SOMA for a year and walked from there through the Tenderloin to my office.