I don't speak for your family history in a personal sense and I get your main point about the ridiculous absurdity of racial groupings, but Islamic states and kingdoms which include the Arabic peninsula, were also notorious slave states for centuries. Again, despite many PC narratives, slavery was hardly the exclusive province of European colonial powers working against oppressed others around the world.

Again though, these modern PC culture groupings that make reference to "white privilege" for all Caucasian looking people fall into the ignorantly ridiculous, or obtuse. I'm Eastern European, from the Balkans, my family and me were war refugees who never had much serious privilege in our lives but I've had people lecture me about white privilege as if I somehow have fault for an arbitrary genetic manifestation on the skin and am thus little different from any caricature of self indulgent wealthy, institutional white business men (not that there's anything specifically wrong with them either by the way).

Ironically, in my direct "privileged white" family, a paternal grandfather was a concentration camp inmate and a maternal great grandfather was an ostarbieter for the Nazis, deported to the Reich during the World War as.... a real-life 20th century slave.