My definition of culture talks about the working values and processes in a company. Some engineers want to be given full problems and be left alone to solve them. Some engineers want to be given specifications on what to build and just turn to code.
Some engineers don't work well in an org without robust automated tests. Others are fine to leave it to QA and toss it over. Some really love agile environments, and some hate them. Some like large organizations where you can be a deep expert in a small piece, and some prefer to be a generalist getting software off the ground.
People have different preferences regarding async communications versus meetings. And even more fundamentally, some people want to go home at the end of the day and forget about work while others want to feel like they're in something so important that they can obsess over. (Most of us are a bit in the middle, but both extremes there exist.)
Culture fit isn't (or shouldn't be) if people like to go drinking at the end of the shift, or if they like to talk about sci-fi or if you fit into the top end of the kyriarchy. (And if that is part of a company's cultural fit, it's going to be toxic in other ways, but as you say, widely marginalized groups and cultures are going to be affected much more here.)