Build scripts often look for system libraries, generate larger artifacts, etc. It's not as black and white as you make it out to be.
https://reproducible-builds.org/
And it should be possible to build the entire system from scratch without trusting any binaries.
https://bootstrappable.org/ https://lwn.net/Articles/983340/ https://stagex.tools/