If you're looking for a reason, collecting $4000 from 1 person is preferable to collecting $1000 from 4 people as there is more work/maintenance in dealing with 4 people. This alone incentivizes landlords for wanting high rents, and a scarce supply, over a large supply and low rents.

In any case, this question applies to any market involving artificial scarcity. Surely you don't think businesses have never tried to make their product artificially scarce in order to make higher margins?