Nix doesn't require everything to be built from source, sure, but everything downloaded must match a provided hash. What's the difference between downloading source code and binaries at that point?
You also forgo any improvements to compiler improvements