But why do this for San Francisco, specifically, instead of just developing another area of the bay that has more space and fewer political issues?

The city has a host of infrastructure and political problems that make things like basic housing development extremely difficult. And while it's fun when you're 20-30ish, single, and without kids the vast majority of employees will bail from SF when they get a little older. I know this seems far off, but it happens to almost everyone. Then... where?