>The ageing and shrinking population is a temporary, one-off problem. There's no fundamental reason why a replacement or below-replacement birth rate is a problem - in many ways it's advantageous - but it's a really big problem over the next few decades because we didn't plan for it.
Taking a perhaps unwarranted broad view, economies of scale allow for us to achieve certain productive activities that we would otherwise not be able to. Taken to an extreme, a tribe of 100 humans would never be able to develop a space program, for example.
Even aside from that, more people means more "black swan" thinkers/entrepreneurs that can have a population-wide impact. It took just one Einstein to develop General Relativity and now humanity will benefit forever.
Obviously humans are chewing through natural resources at an astounding and unsustainable rate, and we'll extinguish ourselves in a few hundred years if nothing changes. If your view is that humanity should co-exist in much smaller numbers in harmony with nature on Earth, your statements are reasonable. I take the optimistic view that we're meant to keep growing indefinitely beyond the Earth, and so a shrinking, aging population is indeed a concern.