Very few software deployment systems make it possible for binaries to be independently reproduced from published sources by the public. AFAIK, it's limited to systems like Nix, Guix, recent Debian, and other participants in the Reproducible Builds project.
However, even within those systems, if you are downloading a compiled binary instead of building it yourself, how can you be sure that you get the "right" binary every time? Does the binary download system periodically "challenge" the binary provider by building from source and comparing with the downloaded binary? If so, does it report its findings anywhere?
It seems to me that even within a software deployment system that enables users to reproduce binaries, you still end up trusting whoever runs the deployment system, because there are no methods of challenging the reproducibility in a meaningful way. The systems I mentioned above sign the binaries, which means that you implicitly trust the holder of the signing key to send you the right binary. But it doesn't mean anything about the relationship of the binary to some source code.
Having said that, if I am using some program by downloading binaries, I am trusting whoever provides the binaries. If I trust them, then a source code audit is valuable to me, even though I can't be sure the compiled binary is related to the source code.