I personally think it's racism and eugenics.
I remember watching a documentary in grade school about overpopulation. When they wanted to convey the problem they always showed the same sorts of shots: seething hordes of mostly black and brown people in places like India and Africa. Meanwhile the scientists and other experts interviewed to argue the Malthusian case were almost entirely white men.
I recall even back then (9th or 10th grade probably) thinking this had a racist vibe, and I'm a white male who isn't particularly hypersensitive to such things. Strip off the intellectual veneer and it was literally a bunch of wealthy educated white men complaining about there being too many poor colored people. Wish I could remember the name of this doc, I bet it would come off even worse through today's eyes.
Of course today the new hotness is to panic over underpopulation and demographic collapse. That's got a huge undercurrent of racism too. Desirable countries can easily compensate for any concerns over population decline by importing population, but the depopulation zealots seem to freak out even more over that. We can't just let people immigrate... because... they never quite seem to finish the sentence but you know the reason.
Population panics just always seem racist. Overpopulation panics are about too many of the "wrong" people, while underpopulation panics are about not enough of the "right" people.