As for zoning, that's likely a chicken-egg issue, but once holding nonproductive land becomes nonviable, reversing restrictive zoning may become far more tractable.
Alternatively, cracking the nut outside SF (and yes, I realise you're in the City) might be an option, where if a sufficient alternate jobs-and-housing magnet were formed, SF would be forced to follow suit. Not my first choice, but I'd take it.
Tackling this as a strategic "having defined our point B, how do we get there from A" question, and finding the least-cost path seems the right approach. Finding examples of other areas which have successfully tackled the problem is another.