You're less likely to make seasoned choices with property. When you buy an index fund, you get everything -- the winners and the losers.
Let's say you have $10M and want to put 20% into real estate. $2M will only get so many properties (a handful?) -- they could depreciate, they're highly illiquid, and you could end up buying in the wrong neighborhood (though, say, adjacent neighborhood appreciated much more).
It's kind of like throwing darts. No real estate agent 40 years ago would've told you that $300k lots with homes are now worth millions, but a mile away those same sized lots are now worth $500k (this is actually true in my area).
The opportunity cost is too high, that you'll make the wrong choice, picking property that appreciates poorly, and had you just stuck with an index fund, you would've done better.