If I operate my own public library and remove a book from there because of its contents it is censorship. It doesn't matter whether I'm totally within my rights to do that (I am) and what exactly the book contents are.

Legally in the US, yeah.

But should that be the case?

That's not true. Free speech as a concept is speech free from censorship, period, not just censorship by government fiat.

Look, you can disagree with that, but I'm not sure why you would. The entire concept is that two ideas, presented before a reasoning audience, can be examined and the one that is better will be likely to be adopted. This is impossible to do if one of the ideas is never presented as it is censored.