After you've downvoted you cant upvote either, so there could just be a flag that says you've done one or the other.
The system serves to record the arrows you've clicked, therefore this information must be stored somewhere. Someone with access to this this information should be able to determine "what stuff an account upvoted," unless the system has been designed to obfuscate the connections(i.e. accounts & clicked items) at the database level.