I think it would be better if the build process ran on both the build server and the devs machine and get the exact same output on both with deterministic/reproducible build processes.

https://reproducible-builds.org/

And it should be possible to build the entire system from scratch without trusting any binaries.

https://bootstrappable.org/ https://lwn.net/Articles/983340/ https://stagex.tools/

Which build server? Not everyone has the luxury, and that also means you have to secure a build server now too. For rust, there is no build server. Packages are distributed as code.