Your party affiliation is a matter of public record, and you're not required to vote for your party. Not really a big deal IMO. Any attempt by the election workers to suppress your vote based on your answer would be trivially detected by a number of security checks.