Keyword: refuse. They can, but pretend they don’t. It’s racism disguising itself as cultural pride. (Source: I’m French and am certainly not proud of that part of my heritage)
It's elitism, France's hate of foreigners is not founded on skin colour or heritage (e.g. both your parents can be French and if your French is bad, you get treated poorly anyway).
But yes, elitism is also present: If you're perceived as French and your french is bad, you will also be treated poorly. But it's a different kind of "poorly". In fact, Paul Taylor talks about that in his stand up "Franglais" (I HIGHLY recommend a watch, it's hilarious especially if you're familiar with both French and British/American culture, and it's available for free on Youtube: https://www.youtube.com/watch?v=Pae2AMnmUVA)