In my experience, it's generally been the case that rich people are anti-consumption, at least on a personal level. Poor people often splurge on luxuries to relieve some of the pressure of grinding poverty. They buy things to signal status in order to attract a partner. A sales tax on luxuries like this is effectively a wealth transfer from the poor to the rich, not altogether dissimilar to government-run lotteries which fund scholarships.
And the Gov sees the rich land owner as the source, not the renter. And it makes the land-owmer seem like their taxes are high - when the burden is not theirs
This argument, followed to its logical conclusion, would tell us to abandon all taxes, all monetary penalties short of total seizure, and any form of finance-based incentive on companies and landlords.
So, what would you propose instead?
Edit: ....Also, this ignores the fact that not all land owned by the wealthy is being rented. Indeed, there are landlords that deliberately keep units empty, in part to reduce supply and keep the price of the units they rent higher. Taxing them even when they're inactive would prevent this kind of perverse incentive and help (at least in some small way) to alleviate the housing shortages in some areas.
This means that a wealthy person owning an empty lot in an expensive part of the city pays just as much tax as the condominium owner next door. This encourages the empty lot owner to build and collect some rents in order to offset the tax. This ultimately increases the supply of real estate and puts downward pressure on rent.