Surely at least part of it is the concentration of wealth and economic activity. There's large masses of decaying rural towns where houses are cheap, but at the same time the whole reason they're decaying is because there just isn't enough economic opportunity to afford even these cheap houses.

I have no clue what the overall balance of housing looks like, but surely it doesn't help that big chunks of the housing market are being deprecated at the same time.