Living the SF Bay Area now, I'd say it's slightly more common to take one's shoes off indoors, but certainly not the rule.
I just generally don't step in disgusting stuff. I don't want it on my shoes, whether or not I'm wearing them indoors. I don't live on a farm. I don't regularly walk across fields or on dirt roads. My shoes are relatively clean, and I don't have wall-to-wall carpet so I don't have to worry about dirtying the carpet so much.