I see voting systems as multiple components. You don't necessarily need to use the government or the existing structure to issue the hardware tokens - I was just proposing that as one potential idea as logistically it seems like it'd be the easiest.
You could set up some decentralized system like the Certificate Authority system that we have for the internet perhaps to verify & issue tokens. That would be fairly difficult, however, as there would also have to be assurance that a single person has not been issued multiple tokens. There would likely require there to be a central registry somewhere.
You can't just "PGP it" in this scenario, as it needs to be one token per voter - and that has to be provable. In a "PGP"-like system - while you can receive higher assurance of identity through other people signing your public key, it provides no assurance against tokens being associated with fake identities, or people owning multiple tokens.
Regardless - identification is just one aspect of voting.
You also need some way to store the votes that have been cast (while maintaining integrity, verifiability, confidentiality) as well as a way to tally the votes (while maintaining integrity, verifiability, confidentiality). That's the component that the blockchain would serve to fill.