Tangentially, if they self hosted using say ZFS would the cached artifacts and other such things be candidates for the dedup features of the file system?
Tangentially, if they self hosted using say ZFS would the cached artifacts and other such things be candidates for the dedup features of the file system?
> AWS S3 - We have been in contact with AWS throughout the week and awaiting a response to meet with the OSS team.
Their current sponsor isn’t a deep pocketed publicly traded cloud provider. All of these cloud providers advocate for open source. Cloudflare could come in and cover these costs and write them off as donations or something similar. For AWS they’d probably not even make a material difference in the accounting software.
I think you’ve got some deep seated angst against the Uber wealthy (join the club) but I never mentioned bezos. I mentioned these companies sponsoring selfishly for the goodwill it would bring.
I’ll try to make my case better next time.
Yes, block-level deduplication would be hugely beneficial. If you're running Nix locally you can set auto-optimise-store = true, which enables file-level deduplication using hardlinks; that already saves a great deal of space.
Its possible to serve a nix cache directly out of a nix store, creating the NAR archives on-the-fly. But that would require a) a big filesystem and b) a lot more compute.
Which are not insurmountable challenges, but they are challenges.
It seems to work well for us, but I am curious how that would look at 425TB scale— would filesystem deduplication pay for the losses in compression, and if so, by how much? How does that change if you start sharding the store across multiple ZFS hosts?
If you were just storing a static set of pregenerated NAR archives, you will not see any benefit from filesystem-provided compression or deduplication.
If you host a live nix store (i.e. uncompressed files under /nix/store), then you could benefit from filesystem-provided compression and deduplication. Also, nix itself can replace duplicates with hard links. But the downside is then that you have to generate the NAR archives on the fly when a client requests a derivation.
That might be worth it, especially since they get great hit rates from Fastly. But on the other hand, that means a lot more moving parts to maintain and monitor vs. simple blob storage.
Then put whatever cache you want in front.
[1]: Ran into https://github.com/aristanetworks/nix-serve-ng/issues/19 with it
in this case, the files being stored are compressed archives (NAR files) so the block-level dedupe that ZFS does would probably find very little to work with, because the duplicated content would be at different offsets within each archive.
beyond that, there's also the issue that they would need a single ZFS pool with 500TB+ of storage (given the current ~425TB size). that's certainly doable with ZFS, but the hardware needed for that server would be...non-trivial. and of course that's just a single host, they would want multiple of those servers for redundancy. distributed object storage like S3 is a much better fit for that volume of data.
0: https://www.truenas.com/docs/references/zfsdeduplication/#co...