That's not the case. Imagine this system: upon receiving your ballot at the polling place, you find n candidates, each with random numbers from 1 to n associated with them. You pick candidate #i, submit that vote, and take home a receipt saying you voted for candidate #i. Then proof is later posted that your vote was counted for candidate #i (using some fancy crypto). Then you can verify that your vote was indeed counted for the correct candidate, but no one (not even you, were you to forget) can tell who that candidate is.
For more information on these sorts of voting systems, see https://www.usvotefoundation.org/sites/default/files/E2EVIV_.... It's intended as a review of remote voting systems, but a lot of it apply to other types of systems as well.