I don't want to erase slavers from history books, only from positions of honor. Stalin and Hitler should be remembered accurately, not honored. Same for other slavers.
It appears to be a valid question, in that slippery slope is not always a fallacy.
I don't think there's really a right and wrong here, it's just the fashion of the times. Who knows who will be rehabilitated 100 years from now and who will be stricken from the monuments :-)