Anyone who does say that is, I submit, either confused, or trying to confuse.
The position, whether you agree with it or not, is "Racism is prejudice plus power". Disenfranchised groups don't have the latter, but they damned well can — and often do — have at least as much of the former as the enfranchised.
EDIT: Or is somehow your position that the prejudices held by enfranchised and those held by the disenfranchised have no meaningful difference in their affects?