We prevent people from using their property as they see fit all the time via zoning. It is in fact mostly those with generational wealth that use this power.
Being told you can't live in your house anymore because it is now zoned commercial gets people riled up, and being told the same because it's now too expensive does the same.
It's a hot-button issue but you can defuse it by putting safeguards in, though they're not always successful - see eminent domain scandals.