Two things: if there is a limited supply of luxury housing, the rich will buy mid-market housing (or low-end) and often renovate to make it luxury. Very common in SF. That means more competition for the lower/middle of the market.

And second, the luxury housing today will be middle of the market in a couple decades.