Property tax isn't just about the income to the state. Property tax is about making sure that people are putting the property to work. If you're second mansion is costing you an extra chunk of change every year and you never visit it you'll be more likely to sell.
Additionally, property taxes are (usually) progressive. In places like NYC where they aren't that should be fixed.