Nobody is saying that only white men have bias. However, white men are overwhelmingly in positions where that bias can be used to discriminate against others. Women may have just as much, but far fewer of them are in positions where that bias comes into effect.