> when the "right" person shows up, how do we know they're the right person?
That prompts an investigation. The “right” person casts an affidavit ballot and the police and courts investigate. If the count is close, the loser usually sure to recount and verify, and any of these incidents then become political kindling. It doesn’t happen because it isn’t worth it individually and difficult to coördinate en masse.