Yes, and it's pretty surprising that the same mechanism isn't used for group affiliation.
The default behavior should be that other people can't release information about you without your approval, whether it's photos, location, group membership, reading habits, etc. This shouldn't be a tough concept for FB.