I think it's downvoting. It's one thing to see (what you think is) a really bad reply to you get some likes, it's another to see a negative number next to your comment, I feel much more compulsion to defend myself for the latter.
Then when I'm REALLY bored I'll go click my own history and see what the peanut gallery thought of my various comments. I get a lot of mega-downvotes for really innocuous stuff for no apparent reason, along with the occasional angry response, so it's best to keep my stress levels down by keeping that stuff isolated.
I'd be a nervous wreck if my phone/watch were buzzing at me every time some dickhead online decides to rip into me. I don't know how people can stand it.
Some niche subs never downvote people, and some downvote almost everything
I think your best bet is to ignore karma altogether on that site
Beyond a certain size all communities harbor this behavior.