>But outside of that structure, what gives someone the right to dictate to their neighbors what they can and can't do with their own property?

The fact that what you do with your own property often has direct effects on your neighbors. This is why zoning regulations exist, for example. And noise ordinances. And building codes. There isn't a magical forcefield that ends at your property line that prevents what you're doing from having an adverse affect on others.

Just like people who don't like noise should be careful about living close to other people, people who like to be left alone to do what they want with their own property need to take the same considerations into mind.