Yup. That's exactly how it goes.
Our entire social structure is based on the notion that economies grow and will continue to grow, forever.
The largest component of economic growth is not gains in productivity, which is what most people think. It's simple population growth. Each additional person means an increase in consumption of goods and services, and as a consequence, an increase in production not only for the sake of that person's consumption but also through that individual's labor. That's growth.
Take away population growth and you'll have anemic growth at best and a spiraling depression at worst.
Turns out that population collapse is potentially very dangerous to the environment. Why? Because a rapid collapse in population would cause economic distress and wars over resources. During economic crisis, environmental conservation tends to (unfortunately) go out the window. And war is devastating for the environment.
Anyway, Brand says the world population requires something like 2.1 kids per person to avoid a collapse in population, but is falling below that in many countries. The green movement of the 1960's was very effective at lowering population growth, averting a potential 'population bomb' that was so feared at the time.
I'm not by any means an expert in this area -- just doing the best I can to paraphrase what so far has been a very interesting and paradigm shattering book.