Would you still have the same trepidation if the data were generally anonymized? I don't think there's a way to say that it would ever be actually anonymous, but it would prevent casual browsing of someone's voting history that way.