Fair enough. Keep in mind, the appeal of San Francisco before all of this happened wasn't that it is vibrant. Different cities can appeal for different reasons.
San Francisco, when people spoke of it affectionately, was the cool grey city of love. It wasn't only that, or always as good as that, but that was the appeal. The appeal of San Francisco is that it was an urban place with a notably less frenetic vibe. I'd also say that San Francisco is actually considerably more urban than almost every other city outside NY in the United States, even though, somewhat paradoxically, it has less of the urban vibrancy you describe.
Here's a link to some urban density maps from a discussion forum.
http://forum.skyscraperpage.com/showthread.php?t=204523
The first thing you'll see is that in the US, it's New York and everywhere else. But of everywhere else, San Francisco has notably more widespread high urban density than the other US cities you mentioned. It's enough that you actually can live that life in SF, even if it is somewhat limited.
This appealed to a lot of people. It actually still does - or would, if people could afford it. Something odd happened to San Francisco. I grew up here in the 70s and 80s, and San Francisco wasn't ever really known as an alpha city for a career. Living here was a softer version of living in, say, New Orleans. Nobody says "I hate living in New Orleans, but I have to be here because of the amazing career options in finance/tech/entertainment." It's not that you can't pursue a profession with ambition in New Orleans, but the main reason people who make the choice to live in New Orleans do so, in spite the problems, is that they love New Orleans.
So it goes, or went, for San Francisco - though San Francisco was a strong beta-plus, perhaps even alpha minus, city for careers. As late as the mid 90s, parents in SF were encouraging the younger set to consider leaving SF for career options, considering SF to be somewhat limited for career advancement. I explained to them, at the time, that tech really was bigger here...
Wow have things changed. Gayle Laackman McDowell wrote a post:
http://www.gayle.com/blog/2015/11/if-you-want-an-a-career-in...
It's hard to explain how unlike the San Francisco I grew up in this post really is. It is the opposite of how SF perceived itself. SF was where you lived because you liked it so much here that you were willing to give up on the career options available in New York, London, even Chicago or Los Angeles. The idea that SF is something you put up with for the career options, but leave if you want an alpha minus career but a better quality of life, is a 180 degree change, and it happened with light speed.
I think this accounts for a lot of the "10 reasons why I hate SF" posts of the lament about SF you see on places like HN so often. Until quite recently, you wouldn't have seen a post on why people like NY so much better than SF that because person would't be living in SF. They'd visit for a conference, say "huh, things sure close early here", and go back home to NY. Any sort of complaining about needing to live here for career advancement would have met with general puzzlement. Even tech, which was bigger here, but not nearly as big a deal as it is now, and mainly concentrated in the South Bay anyway.
I get it, you hate SF but choose to live here because you put career above living in a city you enjoy - a tradeoff that came into existence so quickly it has a lot of long time San Franciscans blinking like deer in the headlights. And SF has largely lost the qualities that people enjoyed about it, even though it sounds like those are qualities you wouldn't have valued much anyway.