It's expected the ones less threatened and in a social dominant position have less fear of hate speech directed towards other populations or even to itself.

A more interesting question is at what point free speech ceases to be beneficial to society and starts reinforcing oppressive stereotypes or to direct hatred and marginalize minority or less dominant groups.

It is, however, understandable when groups that are consistently at risk are more suspicious at unrestricted speech being directed to them or others in similar conditions.

Speech may have undesired consequences.