I'm curious - what were Mission, Soma, and Lower Haight like in 2002?
I used to do a fair amount of work in the Potrero hill area in 2000 or so and dreaded parking there. Then the bust happened and not only did a lot more parking appear, but many of the good restaurants in that area shuttered almost overnight.
We lived in lower Haight until 1999, and it was okay, if you could roll with a little bit of homeless person screaming in the middle of the night just outside the place we rented. Our cars were broken into a few times and we got a few parking tickets a year but that was the price one paid to live in SF. It largely wasn't worth it. First me, then my soon to be wife got a job in the south bay, we moved to los altos to skip the commute.
The funny thing is the big 'displacement' story in the city around 2000 was the gentrification of the Castro, where older more established gay folks were making it difficult for younger, poorer gay people to find places to live near the Castro (which is sort of the traditional gay nexus in SF)
IMO SF is in much better shape nowadays, and it has a lot to do with the thousands of tech companies in and out of the city employing people who add their money and the rest to the city.
The slow rise in tech in the early 90s pushed SJ and SF out of their slumber and downward trend. Now that wealth from tech is permeating the city and bringing economic benefits, non tech people want the benefits derived from tech wealth without the side-effects. Thing is, if everyone of the 6million plus people in the Bay area earned 150k/year, we'd still have the same problems. You can't fit 6 million people in a city with low housing stock and insignificant housing stock coming online. So, some of those 150k earners will do better finances and be able to leverage their earnings to buy --the less financially savvy would be unable to afford SF --and then we'd be back to 'being priced out'.
I've been to the Mission, the Tenderloin, and while yeah, there's the occasionally deranged individual doing the occasionally deranged thing, the situation I saw wasn't anywhere close to what you described, so I assume it's been improved?
It was considered an "improvement" even then! And by "improvement", I mean the whole gentrification and character-of-San-Francisco arguments were bubbling around at all the hip tech parties attended by the VPs of XML Operations of every TomDickHarry.com dotcom!
From my (very limited) perspective, the Tenderloin actually hasn't changed much at all.
Of course parts of those neighborhoods were terrible -- it's difficult to argue that 6th and Mission isn't better now than it was 10 or 15 years ago -- but to define entire neighborhoods by those types of intersections would be wrong. 16th and Mission was -- and honestly, still is -- often overrun with drug dealers and prostitutes, but if you were to have walked two blocks in any direction back then you would have passed countless independent art galleries, rehearsal spaces, and performance studios, not to mention an even greater number of family owned restaurants, grocery stores and shops.
And the best thing about all of these places was their accessibility: nearly anyone could afford to go out to dinner or an art opening, see a band play, or just have a couple drinks down the street from their house. It's that accessibility that's missing now: working class families, artists, and musicians can't afford the neighborhoods they helped build, so slowly, through Ellis Act evictions or sheer frustration, they start to trickle out of the city, taking with them what made the area desirable in the first place.