It's actually much much worse than that. Votes not only have to be anonymous but not verifiable by the voter (i.e. they should have no way to prove who they voted for after their ballot is submitted). You don't want to live in a world where rioters could target anyone who couldn't prove they voted for their candidate or domestic violence for the same reason. Also, if votes are verifiable then it becomes easy for a person to buy votes or for companies to offer 'incentives' to vote for a particular candidate.
Issue everyone a secret key (the digital equivalent of a ballot). They publish hash($key, $name). There are only N values for $name so that can be solved in O(N) time, but without knowledge of $key you couldn't determine whom someone voted for. For the sake of the user you could even pre-compute the hash values so it literally becomes "publish this string to vote for X".
You could authenticate the user by having them give their pubkey at their secretary of state. The pubkey signs the hash like a standard blockchain transaction.
This is a general problem with absentee/mail voting, though, and it's a strong argument. If your boss/spouse/etc is being an asshole there's little to stop them from coercing you to show your paper absentee ballot, marking it in for you, etc. Yes, it's illegal, but what are you going to do about it after your at-will employment is terminated without cause next month? In the privacy of a voting booth, you can vote however you want and nobody will ever know. It's real easy.
I think you can still keep votes secret if you have in-person voting. Just have the secret key never leave the polling place, it's equivalent to collecting the ballot after tabulation (we have paper ballots that are run through a ScanTron machine, we don't get to keep the ballots). You get a printout of the hash values you selected (but not the key) and can validate that the hash exists on the blockchain afterwards (signed by the polling place's private key), but there's no proof as to what the particular hash values actually indicated.
Really though you have to trust that people will treat their electronic ballot with just as much import as their paper ballot. I don't think an electronic works without some kind of secret key/token.