But in the long run, when mortgage rates go from 3% to 6% affordability goes down - people buy based on monthly payments, not the size of the loan.
Of course if wages drastically increase, the affordability issue could be blunted.
So yes, it's not a perfect correlation.