> Until a pattern of bias is demonstrated, such judgments are neither implicitly discriminatory, ...
I mean, by definition they are discriminatory though: you are choosing to discriminate between individuals that you will associate with and those you won't.
This is not inherently bad, but can definitely be unjust toward those on the receiving end even if it's justifiable by the discriminator.
For instance, maybe a neighbor down the street seems a bit creepy, so I tell my kids not to hang around them to stay safe. This is fine. For that person, though, it may be a totally unfair and unwarranted restriction, and they actually are kind and good, and hurt that people won't associate with them.
The point I'm making is that even if everyone is just exercising their rights and being reasonable and safe, the outcome can disproportionately negatively impact the receivers of the (perhaps reasonable) discrimination.