It's odd how being scared of something makes you believe that everyone is scared of it.
There's a significant amount of people who don't actually get scared of "spooky old houses". Just like there's some people who don't get nervous when a giant spider is on their hand. Or when they're talking to a crowd. Or when they're on top of a cliff.
I'd certainly be nervous in a "spooky old house" but certainly not for its ghosts. I do despise spiders for one thing.