>So now I associate that word with all the problematic aspects of the book and assume the person using it is either using the word without having read the book or didn't see a problem with the book when they did read it. Neither really makes me think highly of the person.

You cannot separate the book and person reading the book. Its ok to disagree, its ok to have different views.