Its possible to serve a nix cache directly out of a nix store, creating the NAR archives on-the-fly. But that would require a) a big filesystem and b) a lot more compute.
Which are not insurmountable challenges, but they are challenges.
Its possible to serve a nix cache directly out of a nix store, creating the NAR archives on-the-fly. But that would require a) a big filesystem and b) a lot more compute.
Which are not insurmountable challenges, but they are challenges.
It seems to work well for us, but I am curious how that would look at 425TB scale— would filesystem deduplication pay for the losses in compression, and if so, by how much? How does that change if you start sharding the store across multiple ZFS hosts?
If you were just storing a static set of pregenerated NAR archives, you will not see any benefit from filesystem-provided compression or deduplication.
If you host a live nix store (i.e. uncompressed files under /nix/store), then you could benefit from filesystem-provided compression and deduplication. Also, nix itself can replace duplicates with hard links. But the downside is then that you have to generate the NAR archives on the fly when a client requests a derivation.
That might be worth it, especially since they get great hit rates from Fastly. But on the other hand, that means a lot more moving parts to maintain and monitor vs. simple blob storage.
Then put whatever cache you want in front.
[1]: Ran into https://github.com/aristanetworks/nix-serve-ng/issues/19 with it