He was a slaver. That's at least one thing to hold against him, to say the least. Everything named after him should be renamed, all statues of him in public places should be toppled.
Let me explain my idea on another, more clear cut example. Ancient Greece and Rome have been built on slavery. Almost all powerful and even educated men of that time used slave labour. Should we remove them from philosophy books and rename the months?
Should we just proclaim that everyone who lived then was evil, except the poor and the slaves themselves? But even they didn't hold some higher moral values, they just didn't have the means to enslave other people. So, we probably should say that everyone in ancient world was evil, period.
I don't think there's really a right and wrong here, it's just the fashion of the times. Who knows who will be rehabilitated 100 years from now and who will be stricken from the monuments :-)
It appears to be a valid question, in that slippery slope is not always a fallacy.