Gross inequalities between groups _can_ exist, but that tells us _nothing_ about any particular individual within the group, only that individual's chances _when no other information is available_. But other information _is_ available when looking at an individual.
You seem to be suggesting we somehow theoretically add up all one's advantages and disadvantages- But that means... Considering the individual. Great, I agree. Since we are doing that, phrases such as "recognize your male privilege" mean nothing to an individual male, because it is very possible that individual male has severe disadvantages that make singling out the maleness aspect ridiculous- It would be like telling a minority male shot by a cop to reflect on his privilege of not generally facing sexism like a woman does (setting aside that a "male" privilege here is actually a disadvantage, but you get the idea).
In general people have an obligation to help the less fortunate, I agree.