You're completely overlooking people who would want more children but don't feel they could afford them.
I almost feel like that's self regulating. Scarcity of food results in malnourishment which if it doesn't kill anyone directly, will result in other sickness being more likely to kill people. Same with children surviving to adulthood.