I'd be worried if the population was forecast to shrink below a billion, but that's highly unlikely on any reasonable timescale. I'd be worried if the population rose much above 11 billion, because then resources start getting really tight. Anywhere in the middle is fine by me.
If we do start terraforming Mars, I might be inclined to incentivise population growth. Unless we're absolutely certain that we can make it work, further population growth is a huge risk - every extra billion increases the pressure on resources, which increases the risk of conflict, which increases the risk of an extinction-level event.