Yes, it's about when freedom of speech trumps freedom of association.

If the freedom of association is inviolable, why can't I put up a sign outside my store saying "NO BLACKS"?

If it's about someone making statements at work, would you accept if a business owner said that they'll fire you if you're Jewish, but let you work there as long as they don't know about it?