I think the biggest issue in zoning is the lack of a path to unzone something. If I convince all my immediate neighbors to let me build up (perhaps by giving them some equity in the new building) I should be allowed by the law to build. Today, even if my whole block wants to build up, we cannot.
Personally, I don't think you should have to get universal agreement either. Maybe a simple majority? Life does chance, cities grow to new borders, single family zoning that made sense a generation ago can be absurd later.
As for zoning vs private contract... well, for historical reasons, that's just how it works in older urban neighborhoods. That's the mechanism by which these property rights are decided. It's not ideal. But at my core, I do think the right to live in areas with certain building restrictions is a kind of personal property right just as real as the right to build whatever you want.
This is an excellent point.
Zoning ensures you can buy a nice quite house which is surrounded by other nice quiet houses and they won't be suddenly replaced by high-rise buildings. For most people housing is their single largest asset and asking them to "just move" is not a particularly nice solution
Without legislated zoning this would still be possible but only in gated communities or similar. I'm not particularly familiar with US law in this area but local regulation seems to be a reasonable way to accomplish this.
Where I live a lot of people are complaining about the opposite problem: developers are constructing 7-story buildings in quiet areas, sometimes just outright replacing green spaces.
I personally live in a very crowded central area but still believe that suburban areas should be accommodated.
There are a lot of factors here - property taxes are also sky high for new buyers, so it's a little difficult to consider this one question purely in isolation.
This is still available as a covenant, which encumbers the property deed with restrictions on use. If a bunch of neighbors on a block decide to do encumber their own deeds, they can.