Not quite. Ok, I might be exaggerating with the CS Ph.D. requirement for the average voter, but there are two steps:
1) Verify that my vote was correctly encoded and cast as intended. Every voter must verify this. It can be as "easy" as: printing a hash produced by a voting terminal/device you trust (which doesn't need to be government provided), submitting the resulting encoded vote into a (untrusted) government website where you authenticate with a smartcard, and then later comparing your original hash against a hash published in a newspaper... This exceeds the level of sophistication/care of most voters in any large country.
2) Some organization must check a large trail of signatures, zero-knowledge proofs of correctness, re-encryption mixnets, etc, etc. This can be done well enough as long as every political party and election watchdog has one or two crypto professionals in their employ.
I am not really worried about #2, even though it is harder, I am worried about #1. The important part is that all the checking of the world regarding #2 is useless without at least a representative subset of voters performing #1 (~1% would be enough, but only if distributed uniformly at random, otherwise you can flip votes from the population(s) least likely to check).