Usually, it's the wealthy that tend to leave and depopulation tends to increase "blight" and crime.

But I had no idea the population declined so much, that puts other observations about SF into context.