Starting with 512 bytes of machine code plus source and building an entire Linux distro from scratch without any existing binaries means that it is exceedingly unlikely that such a backdoor would be possible.
This would probably be a bit more difficult to pull off. If someone did pull it off, fully reproducible builds ought to make it readily detectable at least in the absence of some extreme rootkit contortions.
Reproducible builds wouldn't help here, since all binaries would be backdoored equally. They only help with situations where one build machine is compromised but another one isn't.
The much more common scenario would be bootstrapping something like Guix from within a running OS from a much smaller set of initial binaries than you otherwise might. And that host OS could in theory have been compromised. But I think that attack is a significantly higher bar than a compiler binary that compromises itself.