I'd be careful about drawing that conclusion. Wealth historically has been associated with a higher fertility rate.
There's almost certainly multiple factors at play here, but my guess is that cultures where women are less free to pursue careers and where religious values promote childbirth have a demographic advantage. And it just so happens that these cultures are also poorer on average.
If you look within multi-cultural countries you'll find that populations which hold conservative religious values in regards to sex, and population that are less likely to promote women's rights tend to have higher much higher fertility rates.