Yes, but you can commute in it year-round. And with a fixie or single speed, the hills add at least a modicum of challenge.
There's no reason to say that SF proper is too small to take advantage of; I think the appeal is that here, you have year-round, every day exercise. That little bit adds up, just like every day of brutal cold in Chicago or NYC changes you.
That effects people's psyche, and I think that's a major reason why the attitude out here is different.