Your statement that he is terrified of black people is based on you (presumably a black person) running into him outside an event, and him reacting with fear/extreme caution when you approached him?
Not defending Sam, but if that is the case, then it's the kind of thing that Sam can hold up and say "Do you really think my critics are intellectually honest?"
Rock solid evidence is what brings people down. Stretched truths, assumptions, and careful half-truth wording, are all ammo the accused will use to strengthen their side.