That's a distinct possibility, but the wealthy folks you're talking about also had the good sense and economic mobility to move when interest rates were at historic lows, and they aren't likely to be interested in giving up that rate right now.

The effect you described is most certainly real, there's just other pressures to consider.