For instance, infrastructure is a fixed cost. All the investment into roads, power grid, rail, airports, waterways, etc is now spread over fewer people.
Since public projects are usually debt funded, that usually means that fewer people have to shoulder the same debt burden.
When more of a person's income goes to servicing debt, then by definition, there is less for savings and investment. Less investment usually leads to less growth.
And here is where Japan is unique - they have managed to adjust to the debt servicing, and get better at allocating their investments to provide for a growing GDP per capita. Even more interesting is that their growth per capita in the first part of this century outstripped both the US & Germany (http://www.economist.com/node/10852462) - and the US was in a Fed-induced bubble at the time!