In many non-tech fields jobs aren't concentrated so heavily in just a couple of cities. You can decide what you are looking for in terms of climate, commute, recreation, culture, transportation, and so on, find a city and neighborhood that has that, and then go there and find a job.
These people often pick a place to live, with the intent of staying there for the rest of their working life, and raising their families there.
It doesn't necessarily have anything to do with greed when they oppose changes to the factors that led them to be there in the first place, especially when the people who want that change are people who came there not because they particularly wanted to live there but rather because some company they wanted to work for requires its employees to live there, and those people have no intention of staying long term.
Next time I move, I'm going to be looking for a place with reasonably dark sky, good bicycling, a climate that is warm enough that I mostly don't need to turn on the heat, low enough density that if I want to listen to music or watch a movie at 3 AM I won't be bothering my neighbors (and vice versa), a decent chess within 60 miles, and good internet access. You can bet afterwards I'll vote against change that would change any of those, even if the proposed changes would raise the value of my property, because I'd be buying the property as a home, not as an investment that I can incidentally live in.
The problem is more about the Sunset, the Richmond, Forest Hills, Excelsior -- just these huge sweeping neighborhoods which are incredibly underzoned and underdeveloped. Or the people who fight development in SOMA because it blocks part of their view.
I can't imagine how you could make it any more developed. There isn't an inch of breathing space is there? If you built up how would anyone get around? In higher rise places even like London there are parks and a variety of spaces and variation in the layout to get around the extra vertical density.
There are plenty of restaurants and shops, most of which is not yet gentrified, and they just fit into the regular grid as mixed-use urban buildings (restaurant on the street level, apartments on the floor(s) above. There are no strip malls, big box stores, etc.
Like much of the city, developing it further would likely mean building "up", yes. There are also an enormous number of single family homes in the Sunset, some of which could be converted into multi family apartment buildings to increase density.
All that said, I'm not clear on how you'd actually accomplish the above. My family and I live here BECAUSE we wanted a single family home, a backyard, etc. SF obviously needs a vast increase in residential availability, but I'm certainly not the guy to ask about how to make it happen.
Other big cities like London are crowded as well but at least no two streets are the same and there's always some breathing space.