The xz example does not support your case. Not only was every downstream build infected until it was discovered, it also needed a distro-specific modification (to openssh in Debian and Fedora, IIRC) to work at all.
The xz backdoor relied on a discrepancy between the development repository and the released (source) artifact.
While skipping the released tarballs wouldn't have prevented the problem entirely, it would have made it much harder to hide.