I know that it's not Montana, but the vast majority of people aren't walking around in T-shirts in almost-freezing weather. These temperatures are pretty objectively cold, if you ask me.
FWIW, I've been to SF, but have never lived there. I am from (and currently live in) a much colder city. Even during summers where I've been to SF, I've experienced significant wind chills which have forced me to wear a hoodie.