Depends on how private. Speech is communication, which almost always necessitates more than one person, and the other people are perfectly within their own rights to provide some consequences to the speech they hear in private.
Your argument forgets that it's also free speech to react to something objectionable. If the government forced me to do business with you without my consent, that would be compelled association, which is more similar to how Mao's government behaved in the 50s.