When "culture fit" is used to justify not hiring a capable, 43-year-old woman out of the fear that she'll justify the money-making machine that exists when 25-year-old, male, clueless commodity programmers are stapled together into an underpaid, overworked team, the word culture isn't being misused. That's exactly what culture is.
Most companies that have a strong cultural identity have a negative one. It's also not necessary that a company have one. Banks don't trumpet "our culture" but (excluding analyst programs, which are hellish) are decent places to work in spite of the weak cultural identity.
A corporate "culture" eventually realizes that it must defend itself against perceived enemies. And it inevitably ends up being the worst kinds of people-- passive-aggressive, malevolent sorts-- who acquire the position to decide who those enemies are going to be.
Culture also tends toward arrogance, injustice, and hubris as it develops exceptionalism, which is what it will fight hardest to defend. I still have people from Google, three years after I left, going out of their way to fuck up my life because they perceive things I've said as being threats to their culture's exceptionalism. (Never mind that I've been gone for 3 years and have absolutely no power over anything that happens at Google.)
A commercial enterprise like a business will never have a balanced, full-fledged culture. Culture is immersive, not something people participate in for 8 hours per day in order to make money. I would say that, as much as possible, you don't want a strong culture (or, more dangerously, a strong cultural identity) at your place of work. You want people to get in, do good work, be paid well, and be happy. But once you have people start talking about "culture fit" as if it were a real thing, you've hired too many passive-aggressive assholes, and you need to cool it with the "culture" nonsense.