If the population goes down, the system will go through a phase of chaos before stabilising (E.G: retirement systems will default, factories will not have enought workers, companies not enought clients, loan will not get repaid...), and people are very afraid of that. Elon Musk has made the topic quite popular recently.
I would argue exponential growth being exponentially hard to maintain, the chances that the system collapse are the highest anyway. Better that it does with a decline in population than a decline in resources. We can stabilitise and recover more easily from the first, than from the other one.
There were several times in the history of humanity when we here just a few millions. If we plan, we can even control the fall instead of going full throttle.
But we won't plan, nobody wants to vote for the end of the good times. We will protest, fight to keep the growth, and we will crash. We always do.