We go up to SF every 3-4 weeks and find it pretty unnerving each time.
The homelessness (mentally ill), open drug use, and theft/vandalism are pretty draining.
I lived for a long time in Cambridge/Boston and very, very rarely had any similar experience.