It's socially beneficial for people to A) make the place they live reflect their preferences and B) ensure that the ones who capture the positive social externalities of an action are the ones who paid for it.

Being able to make rules about how the area you live is used are a fundamental part of both of those.

Fully externalizing the benefits of an action never has positive social outcomes. No one would bother making their neighborhood nice if they thought that as soon as they did, a million people who had nothing to do with making it nice in the first place would move in and screw it up.