The fact that this is even possible is deeply un-intuitive as it requires some of the most sophisticated cryptography. That's probably the greatest barrier to adoption. When people think of electronic voting, they think about trusting a company to make machines that operate on plaintext, and require humans to guard access to the machines. They aren't thinking about systems that are provably correct, where it is more likely for an asteroid to wipe out the country conducting the election than for the election results to be incorrect.
For the details and tradeoffs, I highly recommend Vitalik's blog.
As others in the thread have said, there's nothing stopping the government from manufacturing millions of fake passport (or even just generating millions of fake passport keys) and using them to rig the election.
>You probably wouldn't want to use the cryptography on the passports themselves to implement the voting system. You probably want to use one of the general purpose zkSTARKs or multi-party-computation systems.
I'm also assuming here that the govt is signing all the passport keys, cause idk how else that would work.