There are political cartoons drawn by fascists from the 1920s that attempt to stave off accusations of racism by pointing out that they include Italians (then show Mussolini as a racist caricature of a black man), as if Italians aren't widely considered white.
If you look at how the British and later Americans talked about the Irish "people" or "race", you'll note shocking similarities to how they talked about Africans, Native Americans, Asians of all stripes, etc etc. Explicit skin color rarely came up.
Racism, especially old timey racism, is all about "how do I define my group as 'good' and everybody else as so bad that I can treat them worse?"