The OP's point holds true. Consider a clean and showered young male that is not dressed like a homeless person. This is generally quite achievable for most of the poor population.
This individual will experience problems if his skin is black that he wouldn't experience if his skin is white.