Not all land in a given area are equal in their attribute and there's fundamental scarcity in desirable lands. The richest presumably purchases houses in locations that has the most amenities, while everybody scrambles for the rest.
Which is not necessarily a bad thing, it is very well possible the lower priced housing has unacceptable attributes, but that leads to different causal factors.
It is also determined by housing supply in a given location and transportation policies, how we decided that homes are investments which increases exclusionary NIMBY behaviors.
It is not merely 'desire to have more' than the other that determine the cost of living.