In many places it is literally illegal to build anything but the most expensive type of housing, due to single family zoning.
How about close proximity to family and relatives?
Sure, you can move to middle-of-nowhere Texas, and buy land for cheap, but how are you going to pay for that cheaper mortgage, if there are no jobs around? Maybe you can get a telecommuting job?