You are right. Some of the sibling replies argue the contrary by using a sleight of hand; stating the desire to associate from only one of the two parties.

Take for example the relationship between a man and woman.

If both want to live with each other, everything is fine.

What if the man wants to associate with the woman, but she doesn't? Do we force her to accept the man? Does he have a right to access her?

Obviously that's silly. We all know that the relationship is only valid if BOTH parties consent.

And yet in regards to neighborhoods the sibling replies act as if only the man's opinion mattered. "That Jew wanted to come in but was refused! People were not free to associate!" They ignore that refusing someone is also part of the freedom of association.