That is the weakness.
It wouldn't just be that their vote would be revealed, though, it would be that then the bad actor could contest the vote, claiming that they were the voter and their vote was counted wrong.
But if people were in the habit of destroying their unique code immediately after the results are posted, then they at least have plausible deniability. "No, I burned that piece of paper yesterday. Can't remember what the numbers were."