It's not about the number of people, but the age distribution. In 2100 half of the Japanese may be over 65 years old, and it definitely wasn't the case in 1920.
In the long run, productivity growth and technological changes will dwarf demographics. I'm not talking huge Mr. Fusion-style breakthroughs, just small improvements that we all see every day. The Radio Shack catalog from 1982 that was posted here a few days ago is a great example. The improvements since then happened so gradually that we don't even notice them until we look back 30 years.