In an election where each vote is counted by hand, it's indeed quite hard and implausible for the government to conspire to rig it.
If, however, the government presented you with a process which had very few points of failure - a single person counting tens of thousands of votes using machines, or mail ballots where all the workers never needed to see the voter, and rely on government supplied information to validate the person exists - these do not require huge conspiracy but only a select few.
It is indeed bordering on conspiracy theory, and the border is exactly on proper process.
Another way to "measure" the security of an election process, is instead of looking at votes gap, look at how many people are needed to conspire to change the results. If your answer is very few, for example, a hacker with access to voting machine, the worker which counted thousands of votes, or the voting machine manufacturer, or the dude with the mail ballots database, it is a possible conspiracy, because it is possible that very few people aren't honest.
A properly designed election process really does require an implausible conspiracy to rig.
An improperly designed election, is almost like a backdoor, and the conspiracy theory defense no longer works.
By the way, there's another side to the coin: in a properly designed election, fraud claims simply do not happen, because it would take hundreds of individual and independent claims of fraud, to be able to claim that the results should be different.
Because the most simple solution, hand counting, works, if the government fails to give you proper verification, it is already the government's fault.
This is also the explanation that should be given in any functioning democracy. You shouldn't need PR campaign or media to establish trust in elections. Well designed democratic elections, by design, the losing side as a verifier can tell they were fair by the protocol itself. The losing side can try to see how many parts of the process they don't trust, and almost always in normally decentralized election, they could discount those parts and the results would barely change.
If your government fails this verification, skips the rational explanation and instead gives you cliche media campaign, and uses an inherently bad process, that's the part where you should already stop and distrust it.
At that point, the government already broke through all the security boundaries of the election process, and it's impossible to know that they didn't exploit it.