well for example there is no law against standing in a public square, be it physical or virtual, and screaming racial epithets. The government cannot forbid it, and within certain limitations (eg "disruption of the peace") they cannot stop someone from doing it with law or force. It is free and protected speech, albeit abhorrent.

However there is no law against our hypothetical racist being uninvited from the opening of an art gallery whose first exhibition is the work of african-american artists. It is a private organization who does not want to associate with someone.

Or to put it simply: it's OK to downvote someone. That's how communities work!