We need maybe like 100 million or less people total on the planet. And 7 or 70 or heck 700 billion littered through the rest of the solar system. The chances of the survival of our species and life in general on the planet the would be infinitely greater than what we have done in the past 300 years. Population growth outside the planet raises absolutely no questions about the survival of the planet or species. The rest of the solar system is already completely and utterly dead as far as we know. No arguments about pollution or stability or sustainability or survival apply to it.
The rest of the universe as far as we know is our canvas, but please spare the damn planet from where we came. It's rare and nearly impossible to replace.