I get it, there are real issues. But minimizing real concerns of people that are the stakeholders isn't fair. Construction means years of noise and dust, and traffic issues in many cases and people don't want that. People are wary of the character of their town being ruined - what makes the town a great place to be. I agree that there's usualy a middle ground that could be found. But importantly, to your last point...
> It's fine to not like the design of a particular building, but to enforce design aesthetic onto someone else is also a failing of zoning.
I disagree. Go to somewhere like the UK and you'll see in many places they not only restrict what you can do to existing structures, but they dictate which materials can be used to build new developments. They do this because it allows the area to develop while also hopefully maintaining the character of the place. The thing that makes the place nice today. If someone wanted to put up a house covered in vinyl siding, they wouldn't be allowed to.
I fully support enforcing new developments having to use certain materials and be restricted to certain sizes, styles, and layouts.