Jumping from generalizations about groups to specifics of individuals is inherently unjust - as is the inverse, generalizing about a group from a few individuals. Generalization from systematic group aggregation is useful for exposing systemic biases, but action should be specific to the circumstances of individuals independent of group membership or the risk of injustice is very high.

Unsafe generalization is at the root of prejudice. Racists / sexists / bigots generalize from the worst instances of individual behaviour to a group, or aggregate statistics about a group, and then apply the generalization in specific individual scenarios. A crude example, taking the generalization "Jews run global finance" - and it's true that they have been historically over-represented - and then applying the generalization to specifics: "you're a Jew, I don't like Jews because they run the world".

A good rule to bear in mind before leaping to prejudice is that the variance within groups is larger than the variance between groups.

White privilege is a prejudice concept built along the same architecture as racism and sexism. It takes aggregate group attributes and tries to enforce it in the particular against individuals. You've just rehearsed the line yourself - "you benefit from racism" - you've given an example of instantiation of a group attribute upon an individual without evidence. It is literally prejudice, and it's unjust, even if it's more likely to be true than false.