As a Jewish family in Germany (currently living in Berlin) I wouldn't want to raise my children in a small eastern community where they will be the only Jewish kids in school (I get the impression in North America anti-semitism is a pretty fringe phenomenon, but it's still "a thing" in Central Europe).
And similarly I probably wouldn't have wanted to be the only Muslim family in your town.