I think what you say about downvotes is true. Eg a comment will be popular one minute, and then suddenly it's not. I also get the issue that my overall karma goes down, while recent comment values stay the same - which could be malicious downvoting of historical comments. So, I think something odd goes on. I comment a lot, generally thoughtful comments, and yet my total karma over the past year is way down.
I'm interested to hear what you think it could be. As users we have no visibility of downvotes, who gave them etc. You say bots or people - do you have any reason for thinking this? I'm interested to hear more info, if you have any thoughts.