It has to do with the concept of "structural" racism, as opposed to plain bigotry, which can be done by anyone. Structural racist aspects are related to society/institutions, which is shaped by the majority.

While I understand the critique, I disagree with the academic terminology, and think insisting on it leads to misinformed intepretations of others' arguments, like saying "black people by definition can't be racist".