That's what the racists think too, here in the U.S. anyway. They think they are just not okay with the "culture" of the people they are racist against.
Can I not be okay with someone's culture without being racist?
I don't hate or discriminate against Indians, but I despise how they view and treat women in their culture.
And what about the people who have distanced themselves from that culture (if they even experienced it at all) but are still that race? You can differentiate based on the actions and ideas because those are under the person's control for the most part, and not by protected categories because those are not. Despise the people who treat women that way, which might happen to be statistically applicable to a race, but not the race wholesale.
I can appreciate your logical reasoning, but people will continue to make snap judgements based on the knowledge and past experiences available to them.