You can envision a system as complicated as you want, but it'll always have some sort of flaw because, well, bits are bits. I've had this discussion with many people and the same solutions arise again and again... but most are useless.
You could get the source code for the voting program... what if your compiler is backdoored? You could compile the compiler yourself, with which compiler? The backdoored one?
And even if the compiler is trustworthy... who's guaranteeing that the real binary is installed in the machine? You could download it in an USB drive and check yourself or get a hash from the machine but... how can you be sure the machine is reporting the binary that's actually running and not just a fake reply? Could you trust the reported MD5/downloaded binary from the machine?
Easy! - you say - You could upload the binary yourself to the machine! But how would you be sure you're running the uploaded binary and not a fake one?
And what if any of the machines inbetween are rigged?
Then cryptoguys come in: you could digitally sign your vote (forget secrecy) with a state-issued certificate (we have those in Spain with our NID). Maybe encrypt it so that only the far end could read it. You'd then check if your vote's been cast fine online. But how'd you be sure you got the real result?
Once the data left the machine you have to trust that everything's fine on the receiving end. What if the vote count just ignores votes and just does whatever it wants?
And the worst of all... how's your granny supposed to check it herself?
There are some advanced cryptographic techniques relevant to e-vote (some are quite clever) but even the best cryptographers will tell you: do not trust electronic voting!
My advice is: keep with pen and paper! It's easy to tamper with too, but more people know how it works and it's easy to see the inner workings compared to bits over the wire.