I downvote because I have been personally offended by something that has been said. Most often I have been offended by the sin of bad taste. This is, granted, a pretty wide area, but when somebody writes something egregiously distasteful—whether it's a terrible sense of humor, or a misinformed righteousness, or craven pandering to the self-important 'social media' upper crust, or indeed the myopic, worthless blather of said upper crust—I am aesthetically offended by that sort of thing. It offends my sense of decency, my sense of wit and interestingness. I suffer in the same way as I do when I witness incredibly stupid people on television, or very bad music, or indeed when I find myself buttonholed into a painfully tedious conversation with someone who is similarly lacking in good taste, be it social, intellectual, aesthetic or otherwise. I suffer and therefore I downvote, in an attempt to exorcise my pain.

Maybe that would be the easiest rule of thumb, then. If you say something which would, if I were chatting with you at a cocktail party, make me fearfully look for some excuse to go refill my drink, I will downvote you.