The assumption that your stock price necessarily should go up I think is an assumption that will change somewhat in the next 50 or 100 years. In this sense, the rich, those with pension funds, but largely the financial managers dictate a lot of what happens in the economy.
Let's say you have $10M and want to put 20% into real estate. $2M will only get so many properties (a handful?) -- they could depreciate, they're highly illiquid, and you could end up buying in the wrong neighborhood (though, say, adjacent neighborhood appreciated much more).
It's kind of like throwing darts. No real estate agent 40 years ago would've told you that $300k lots with homes are now worth millions, but a mile away those same sized lots are now worth $500k (this is actually true in my area).
The opportunity cost is too high, that you'll make the wrong choice, picking property that appreciates poorly, and had you just stuck with an index fund, you would've done better.
The index fund is the greatest thing to ever happen to casual investors, but with millions of dollars at your disposal you should no longer consider yourself part of that group. They're also not infallible, in the long run they're up but in the short run you can lose a lot of money in them, just like any investment.