Landlords, whether small or large, rely on rent-paying tenants for income, and their existence does not decrease the supply of housing. Restrictions on building do.
With them holding the property, they push prices up for buying, which forces people into renting. It's a vicious cycle that just encourages them to buy more property. At some point the whole market may collapse, but hell, they may get a bailout.