To insist that the "prejudice + power" argument is premised on bright-line "race == power" is to mischaracterize it to the point of being unrecognizable.
It's about relative power. If I'm black, and your boss, and have a prejudice against your race, I can be racist. If I'm black, have no meaningful kind of power over you, and have a prejudice against your race, I'm just prejudiced.
That's the position. This "ALL" lunacy is a multiple fallacies in one, and it remains a bitter disappointment to see a crowd of people who otherwise pride themselves aloud on being so smart, buy into it so unexaminedly.