EDIT: I should add that my point is, there's no reason it couldn't remain this way forever. Urban revival could be over, even though the desire for it isn't.
As an aside, I'd say people satisfying their desire to have their own space is great thing that we should be optimizing for.
The sad part is the inflated rents don't seem to go towards these externalities either, but to landlords' pockets and land managers' margins. Even though the price is theoretically correct, the market reflects it for the wrong reasons giving the sell side an undeserved moral argument.
Rent in SF is going down actually. 2 years ago we moved into our apartment with my girlfriend and whatever price they mentioned our immediate response "YES WHERE DO I SIGN" just because we were so happy to even find a vacancy at all.
1 year ago they wanted to raise our rent $400 (about 6%), we negotiated it down to a $40 increase. Pretty good right?
Well, this year we negotiated our rent down by almost $300. In SoMa. Our rent is now cheaper than it was when we first moved in.
Next year we're hoping to move to a bigger place for the same price or cheaper. It's probably going to be possible.
Fun fact: Our apartment building considers it a success if they get 50% of their tenants to renew the lease.