Having un-utilized land in your family generation after generation codes as noble, but economically this is much worse than owning an AirBnb or a vacation home or being a Chinese investor who collects vacant luxury condos, or any of the other boogeymen of housing discourse. These people hold one unit off the market, and in the meantime they maintain it or finance its construction. The landowner builds and holds wealth for nothing but parasitism.
Being told you can't live in your house anymore because it is now zoned commercial gets people riled up, and being told the same because it's now too expensive does the same.
It's a hot-button issue but you can defuse it by putting safeguards in, though they're not always successful - see eminent domain scandals.
But this is more of a straw man than a real objection; as there are various and sundry ways to resolve this (for fixed income, you can have property tax deferral until sale-after-death, you can have homestead exemptions, etc etc).
I fail to see why that’s a problem. Those taxes go to fund local services (like schools, police, fire, etc.) whose costs likely scale when the local cost of living and land values scale. Buying land doesn’t mean you should get to lock in the price you pay for these services for perpetuity.
Similar arguments would go for rural municipalities that might want to encourage multi-generational agricultural living, for example, if that's what you're getting at.
If anything a pure land value tax should be more predictable than a property tax, I got hit with a major property tax increase and looked into it, the town had calculated the new property tax based on an incorrect square footage and number of bedrooms for my house. After filing an appeal and going to court I got it fixed, but that was a more capricious process than if it was simply based on the value of the land under the house.