I'm no fan of slavery, but far too many things are named after Columbus, Washington, Caesar, Alexander and the others in history who have taken and held slaves.
It appears to be a valid question, in that slippery slope is not always a fallacy.
I don't think there's really a right and wrong here, it's just the fashion of the times. Who knows who will be rehabilitated 100 years from now and who will be stricken from the monuments :-)