It's such a common assumption that when you imply that the children of wealthy African immigrants might not have any more insight into the culture and experiences of black US descendants of slaves than the median US white person, you're reflexively accused of being racist. It's racist to believe black people are fungible. We wouldn't expect the child of an Italian diplomat to give us insight into the experience of growing up on Staten Island.

edit: tbh, it's usually a pragmatic belief. It's easier to convince upper-middle class black people that what you want to do is good for black people in general (because their culture largely overlaps yours and they share your values), and wealthy black people and average black people look the same in pictures.

edit2: I hate to come back to this, but just look up Ray Nagin and his response to the hurricane that ultimately washed all of the black people out of New Orleans for an idea of the character of a lot black leadership in the south, and its total disconnection from any working class awareness. Just looked him up now to see what happened to him since and was not surprised.