I dunno... I barely even notice the "who" element when I upvote something. I upvote purely based on how interesting the link is.
I do think there may be an element of "preferential attachment"[1] going on in regards to how upvotes get distributed, but I can't prove it.