The high birth rate communities also tend to lack the technological acumen. Of they take over the population, medical, agricultural outputs would decrease. Amplify that a bit and they will likely control themselves well through starvation