> Basically, in my ideal world, whoever builds an ugly enough building should be liable to remove it or improve it if it is deemed too ugly by a majority

This sounds like the words of a “community oversight” committee obstructing the construction of housing, and we already know the effects of that on the housing market. Society has swung too far into allowing other people to tell someone what to do with their property already.