>Should community residents not have the freedom to enforce restrictions on their own communities?
Definitely not. Land is a common asset of all of society: taking away building sites while basically all urban areas (ie: with jobs) suffer housing shortages is exploiting the commons.