If you have a known, good state, then you don't have to. You simply use the source or binaries in the repo after checking hashes or signatures on a few mirrors. If your state is that bad, then you're already compromised with possible a rootkit.
"Even if you have all that, how can you be sure no one tampered your system at some point?"
That takes way more than reproducible builds. You need source code for a compiler that doesn't have 0-days in it in terms of backdoors, optimizations that eliminate security checks, or passes that create exploitable faults. That's called a verified compiler. Only one exists for C (CompCert) that's proprietary with quite a few open ones for ML languages. The SCM's distributing that source code must be secure so they aren't adding modifications to source or your OS binary you started with. During transfer, they have to be signed or the transport has to be secure. You then need some kind of local tool you can trust to bootstrap it. If you trust nothing, you're going to be using ancient hardware you bought in person with cash to run an interpreter you wrote by hand that executes something equivalent to a small C compiler you use to compile the others that you inspected by eye or trusted via 3rd party review w/ hash/sig confirmation.
If exotic attacks like Karger's are in your threat model, so are 0-days that are accidental or deliberate along with endpoint and SCM attacks. Reproducible builds don't protect you from much in that model. The stronger methods, from endpoint protection to SCM to local tooling, will pay off over time in more situations given they're independently critical. It's why Karger et al put them in first standards for INFOSEC to begin with.
The other problem is that making all binaries come out equal creates more of a monoculture that aids attacks on endpoints. Many good tools in the field automatically harden C programs by making them memory or data-flow safe. Softbound+CETS comes to mind. Others obfuscate the hell out of them on a per-user basis to reduce odds of one-size fits all attack. You can't use these techniques with everyone having same binary. So, it's a step back from existing compiler and system security methods in total assurance you can achieve.