The question is really: do you care that there is more wealth overall, or do you care that you personally are better off. Most sane people have no urgent desire to start cramming more people into their neighborhoods. What people want are higher standards of living, not the same or worse standards of living split among more people, even if the overall economy is 'bigger'. Many third world population booms are decimating quality-of-life, especially in places like Nigeria, India, and Brazil. The only people who benefit from a bigger overall population and economy at the expense of individuals being better off, are the people at the very top.
They want a bigger pie because then their companies have bigger markets to sell their goods. If it comes at the cost of increasing the population so that everyone else is individually poorer, they don't care. You can't become a billionaire in a country of only 100,000 people. There aren't enough people from whom wealth can be concentrated.