I'm not here to defend unnecessary zoning restrictions, but your argument here is that the right to not be affected by the externalities of someone else's decisions should be limited to the uber-wealthy?.
It's completely reasonable to believe that a given person should be capable of quiet enjoyment of their property without having to purchase everything remotely close to it.