I disagree. Much of Manhattan is now
cheaper than SF, and it faces all of the same challenges. Heck, Manhattan doesn't even have the benefit of being connected via land on one side, and its total area is considerably smaller. Getting
anywhere off Manhattan requires crossing water.
But yet its real estate availability is much higher than SF, and its commutable feeder cities/suburbs are innumerable.
The fundamental problem is political. San Francisco is a desirable place to live - tech industry or otherwise - but everyone wants in, and once they get in, they do everything in their power to ensure no one else gets in. It is literally the most development-hostile city I've ever seen.
San Francisco is expensive because San Franciscans would rather starve, see people evicted, see an exodus of the people and culture that made the city famous, suffer incredible transportation problems, etc etc, than lose a single dainty Victorian house on their street.
Forget sustainable development. Forget development that aligns to existing neighborhood characteristics. Politically speaking, the only development plan on the table in SF is no development whatsoever.