Say that he's got $500K in cash but can put only $3.8K/month toward the mortgage, while the average person around him has $250K but can put $5K/month toward a mortgage. Average housing prices are $1.25M, so at 4.5% interest rates he's competitive. He puts down $500K and borrows $750K @ $3.8K/month for a $1.25M house, while the average borrower puts down 20% ($250K) and borrows $1M @ $5K/month.
If interest rates go down to 2.5% like they did now, the average borrower can now borrow $1.25M, so home prices go up to $1.5M. His cash stash still would require a $1M loan, which would cost around $4K/month, so he has been priced out.
If interest rates go up to 20% like in 1980, then the average borrower can only borrow $300K to maintain $5K/month, so home prices drop to around $550K and he can purchase with cash, or a tiny mortgage. All of the homes between $550K-730K (previously, roughly $1.25M-$1.7M) are now unaffordable to the average mortgage buyer, but are well within OP's range.
Inflation acts as a tax on savers and a subsidy for debtors.