I don't like jazz. I think that a lot of it is self-indulgent noodling.
I can appreciate the talent of some of the artists, but I can't get into it. I feel the same about country music and rap: there is artistry there, but I can't stand 90%+ of it and don't want to waste my time listening to it to find the really good stuff from the derivative, self-indulgent crap.
Yes, I pay attention to the lyrics (because of my time as a vocalist), but I also pay attention to the music.
I think most people don't like 'jazz' because it's an ill-defined "style" that ranges from reasonably well known swing and "standards" to obscure fusion or acid jazz. The more obscure you get, the more inside baseball the discussions become and the less accessible they are to people who might just "like" the sound of a particular song. (The same could be said about the more obscure corners of 'rock'.)