Cities with poor quality of life and high cost of living are that way because their laws and policies aren't effectively managing the high population. If you move to a low population area, the problems you encounter are wildly different (and yes small towns still have big problems to solve). And if you move to a city that has a large enough population that SF's poor city planning actually matters, you're going to have to move a lot of people to overwhelm the status quo. And if a small number of new people dramatically changes the feel of a city, well that sounds like you had 51% of the people who wanted it one way overriding the 49% who absolutely hate the city they live in.
This fear feels like a socially conservative "I don't want them folk coming here and changing things". If new people move in and things start changing, they were always going to change and a lot of people have been waiting for that change for a long time. Embrace it.