In most cases, I'd agree. But when my flagging corresponds to assaulting someone who many around these parts see as some kind of folk hero, I'm not surprised when some downvotes come simply from people annoyed at me for calling Assange out for being a publicity whore.
That said, my guess is any upvotes are simply people agreeing with me as well.
One reason for the downvotes may be that you basically just called him names, rather than making a reasoned point.