The density of cities like Manhattan/SF mean you have to have some level of regulation above and beyond what makes sense when everyone has an acre of his own land, to do with what we chooses. Whether it's noise, pollution, traffic safety, fire codes, etc., I think giving up a little bit more freedom than you would living on, say, a farm, is the price of living in a place with so much jammed in to so little space.
Maybe something to think about, if you take umbrage with people "imposing their whimsies" (I'm not being sarcastic, I really think about this a lot). Austin is nice, but it's a car-city. Show me one walkable city without the petty tyranny of city government and I'd move there pretty quickly.