Though I was actually surprised it was so high. That's units, not bedrooms, and it's only a city of ~800k people. I guess there's a lot of 1-person households to bring down the average despite all the families and twenty-somethings in 2-4 bedroom units with at least as many inhabitants as bedrooms.

Many of California's cities have laws requiring massive, incredibly wasteful amounts of parking to be built even in supposedly high-density or transit-oriented communities. In addition, there are rules saying houses have to be set back a certain distance from the road, which takes a good deal of space. Finally, most places also require that roads meet a certain (high) "Level Of Service", meaning lots of cars can move through very quickly. Nevermind the fact that by building things like this you are forcing them to go much, much FARTHER. Have you noticed that driving through LA (or San Jose for that matter) is mostly driving through a sea of parking lots and wide streets? See http://en.wikipedia.org/wiki/Level_of_service for more.

There is also a cultural preference for lots of land; I'm bemused by how many people mock "McMansions" for having small yards but never spend any time in their own yards.

This is less true in the bay (and not at all in SF) than other parts of the state (looking at you LA) but it remains a problem. Also, areas where the majority of the population are homeowners tend to vote against new housing construction. This increases the value of their own homes at the cost of strangling newcomers to the area and making the cost of living exorbitant. Of course, without new housing those areas will continue to consist primarily of homeowners, so the cycle continues. Even in supposedly lefty places you see people oppose construction that would help increase housing supply, presumably because it's bad for the environment or will damage neighborhood character. I'm sympathetic to those claims, but making people commute from 40 miles away (if they're LUCKY it'll be on BART or Caltrain, but probably not) can't be good for those things either.

I wonder if it's time we reconsidered whether it makes sense for highly interdependent neighboring communities to be separate cities. Would we have a more integrated and efficient transportation, housing, and office infrastructure if the bay from San Mateo up to Mill Valley and east to 680 were the same city? Quite possibly. It would at least mean, to some small extent, that they couldn't say "we'll take your companies but housing the plebs is somebody else's problem - only huge single-family homes for us, thank you!".

The problem is that getting around in that part of the Bay Area still requires a car, so units with no available parking are much less desirable. Having 1 parking spot per unit, and 1 level of parking, effectively limits the height to ~4 floors of marketable residential units (a second level of parking would be more expensive than the first, and allow for ~8 floors of marketable units).

People don't cram into arbitrarily small spaces, so one thing to remember is the cause and effect: way more than 840K people want to live in SF proper, but there's no room. So 376K housing units effectively sets the population cap. But because a larger pool of people want to live there, the prices all go up. Rent control fixes that for incumbent residents but worsens the problem for everyone else. The only way to alleviate that is to create more houses so that a larger percentage of the people who want to live there actually can.

There's also a massive amount of rental apartment stock at any time. This means that you have the liquidity to make any tradeoffs necessary, by balancing your budget, commute distance, number of roommates, and the amount of space you need.

New York is also connected to several public transit systems (NJTransit, Metro North, PATH), so you can often go way cheaper, by leaving the city bounds itself.

It's not all sunshine and roses, though. Rent is going up in a lot of the previously-cheap neighborhoods in Brooklyn. This hasn't really affected me, but my sister has been priced out of multiple apartments in the 5 years she has lived in the city.