Whenever the master branch and slave branch thing comes up people say that it's fine because it has nothing to do with slavery.

The thing I don't understand is, even if it had something to do with slavery, why is that harmlful?