For the record, I am not saying it's not skin color. Primarily it's race/nationality (=~ skin color), and thus british empire falls on the same side of West/Europe/Caucasian people. What other reason is there for the people of west to not treat the atrocities that colonial empires committed (towards brown/black/native people) the same as what Naizs did to white european jews? Especially given what the colonialists were an order of magnitude more cruel in both savagery and the number of people murdered?
As with any puritanism movement, there are other factors determining "purity". Aside from skin color, you have religious denomination.