Not defending Sam, but if that is the case, then it's the kind of thing that Sam can hold up and say "Do you really think my critics are intellectually honest?"
Why? It sounds like they were in an environment with many people and Sam reacted negatively to the black guy. It's not like the story was, "so I followed him down a deserted alley and he got scared, so he must be racist."