121 karma · joined February 13, 2022
If Git repos were routinely terabytes in size, this would be a much harder problem. But the vast majority of repos are <1GB, and at that scale caching dominates. Even with imperfect locality, you can cache enough of the working set that those reads don’t turn into s3 round trips.
In a low-latency environment, I haven’t found filesystem round trips themselves to be an issue for Git workloads either.
Works pretty awesomely with git workloads, even Linux sized trees.
That can literally be the stack, but the same argument could be made about object storage itself: it ultimately writes data to local disks, so why not write files there directly?
The value of object storage isn’t just its API. It’s the durability, distribution, failure handling, and capacity behind that API. ZeroFS lets POSIX applications use those properties. Its server and cache are replaceable and the bucket remains the source of truth.
If one machine and its disks are enough for your data and reliability needs, then yes, a local filesystem is simpler. ZeroFS isn’t intended to improve that case.
zerofs mount has durable fsync and fcntl byte-range locking.
Inotify works for changes made through the local mount. It won’t report changes made through another client, though.
Small writes operate on 32 KiB extents. Partial overwrites are read-modify-write, but the FUSE writeback cache helps combine them, and the resulting extents are packed into segments rather than producing one S3 PUT per write.
I’d be interested in the results if you try it.
There’s no deduplication, either whole-file or block-level. That’s intentional, mostly because of the impact it would have on locality.
If by CoW you mean reflinks, those aren’t currently planned either. They avoid the content matching part of deduplication, but still require sharing extents between files and come with similar locality and complexity tradeoffs. Internally ZeroFS is copy-on-write, with immutable segments and checkpoints, but that isn’t exposed as reflinks.
stat doesn’t pull the file contents from S3; it only accesses the metadata tree, which is usually cached.
I haven’t benchmarked Borg or Restic specifically. Sequential writes can comfortably reach several Gbit/s. For follow-up runs, if they only stat unchanged files, that should stay entirely in metadata.
The default Redis/Valkey configuration should work fine for conditional_put. NFSv4 is unlikely for now. It would add a lot of surface area, and I’m pretty happy with where the 9P extensions are today.
CI also runs pjdfstest, xfstests, stress-ng, ZFS scrubs, and Jepsen crash/failover tests: https://github.com/Barre/ZeroFS#testing
ZeroFS is still pretty young as storage projects go, so I completely understand wanting to see it prove itself over time.
It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.
The mechanism is explained here: https://www.merklemap.com/documentation/how-it-works
It’s a proxy that does just this.
It can be easily configured to block any host or path.
It does not mean that there is not a single bug, but I do not think it is fair to completely discount this approach. Especially when the alternative is browser extensions which bring their fair share of trouble regarding trust, performance, limited capabilities or even security.