"Not a culture fit" could easily mean something like "this person and I have a very fundamental (probably irreconcilable) disagreement about something important". Fundamental disagreements are probably going to result in disagreements about things that emanate somehow from the more fundamental thing.

For example, some people seem to think that when they go to work, they do so voluntarily as part of a mutually beneficial arrangement with their employer. Some people think something more like that they are forced to work and are essentially a slave. These two people will probably have disagreements about ideas that are further from the root than their idea about whether they are essentially free or slaves.

If I were looking for someone to work with, I would be looking not just for the necessary skills, but for deep compatibility. You seemed to be saying "not a culture fit" is just a way of rejecting people without seeming racist/sexist, but I think it's pretty clear that's not true.