I totally agree that "people living together in a small area" can pretty much do what they want. The only issue is when that starts to screw over other folks, and when all the other small areas get together to do the same.
Then it's pretty reasonable for the state to go "you can't just never build houses. That's a decision that screws the rest of the state" and enforce that decision