Homeowners benefit from the ability to obstruct/delay/deny others from building more housing, because it makes them a frighteningly huge amount of money by making their property worth more.
Morally yes, these people ought to be told to go pound sand; the very fact that there's legal authority to prevent dense/tall housing from being built is abhorrent, but it's the function of those boards (and everything else is secondary) -- to "increase property value" by artificially restricting what other people can do with their own property.