The idea is basically the same as any insurance system: the bigger the pool of people participating, the more "shocks" can be absorbed without affecting the long term viability. If you have a tiny pension fund for just one person (e.g. your typical 401(k)), if it has invested its value in a way that makes it impossible to fund that person's retirement, then that person is totally screwed.
On the other hand, if you have 300 million people with upcoming retirements spread over a 45 year timeframe all as part of the same pool, the system can absorb short-term catastrophes without affecting the ability to pay people their expected pensions.
But you only get this by having a coordinated, centralized system. You do not get this by having every individual invest in the stock market, because there's no way to shift money from those currently well-ahead of expectations to those currently needing payout. There's also no way to "borrow" from the capital assets in the expectation that over a period of 30 years you'll be able to repay.
All of this was broadly understood in the 1930s-1980s. But then people were mislead to about the benefits of individual investing, of how their 401(k) would pay out more than the company or state pension plan. While that is certainly possible, it is also unlikely, and the side-effect is that the risk exposure to individuals is substantial. The only winner has been the financial services sector, which has made out like bandits on fees/commissions etc from managing these funds that used to be locked away in corporate/state pension organizations.