I dislike him because of his stated, explicit bias against LGBT folk, and his frequent rants about the "gay agenda". True, I think his Mormonism motivates these things, but many if not most Mormons feel differently (I have no idea the exact statistics), and were he still Mormon and not a homophobic bigot, then I wouldn't have a problem with him.
I also liked Ender's Game as a kid. I like it now. I just think the author is a jackass.