Here's a good example of the snobbery associated with this book. Implying it was "difficult" like it was over my head. It wasn't difficult, it was tedious, boring and meandering.
If this is what you're interpreting as people talking down to you / claiming that you're too dumb to understand a smart book, you may want to consider that it's a you being overly defensive thing. You're allowed to like what you like. If some people have been dicks to you about your taste, I don't see how that has anything to do with this book. Forget those people.