Sometimes it really means you have a specific culture you want to maintain. But you better watch out if you do this, at some point, all the "yes people" in your culture will end up missing something that would be obvious to someone who doesn't fit your culture.
Basically, if you hire for "Culture Fit", you're saying that diversity is bad...