Couldn't you just continue to serve the tiny .narinfo files from a trusted central source while letting bittorrent or ipfs cache the .nar files (big, but can be validated by hash)? It doesn't allow for untrusted builds but it would work with all kinds of derivations.
In principle, signatures also address the cache itself being untrustworthy, right?