Hacker News Reader

Comment by bmurphy1976 | Hacker News Reader

bmurphy1976

Question:

I currently run ZFS via Ubuntu on my home devices. I've had a very positive experience with it. My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time. It's been fantastic for all my other devices, particularly the snapshotting features and ease of which I can ship the snapshots between the devices.

I've read some comments on various forums that seem to hint Ubuntu is moving on from ZFS (see https://discourse.ubuntu.com/t/future-of-zfs-on-ubuntu-deskt... for one example). This is obviously concerning and the lack of support across the whole Linux ecosystem continues to be a nuisance.

Realistically how well could bcachefs serve as a long term ZFS replacement? I know very little about it. I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go.

View on HN

I've been a storage engineer since 2000. The industry rule of thumb for file systems is to give them ten years after mainline release to bake. And the longer I've done this, the more I trust it.

Mainline release in the kernel or actually be used by distros?

10 seems a bit short ;)

i switched back from btrfs to ext4 because i still had problems with it

> I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go.

What are those issues? I've not followed the conversation, but I've been running btrfs for over a decade on multiple machines with different usage patterns without any issues.

I'm genuinely interested. Are those substantiated claims? Or just the general noisy rants you can find about essentially anything people have opinions about?

In my own case: Btrfs performance got absolutely terrible once I was up to a few thousand snapshots. Never had it lose data, but I also never ran it long enough to lose a disk.

This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots.

I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better.

I don't blame people for having bad experiences with btrfs, it is very inconsistently documented.

I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.

My setup is that I only have subvolumes for root, and (for now) my Doom Emacs config. Whenever I do a system upgrade, or a Doom Emacs upgrade, I manually take a snapshot of the appropriate subvolume first, just in case anything goes wrong. And then I only keep a max of 7 or so snapshots before I go through and delete the old ones.

Has worked well for me so far and saved my butt a few times, and since I don't have a bunch of subvolumes to manage, I'm not overwhelmed. It's nice and simple, and I need nothing other than the default `btrfs` CLI utility.

That's what's great about btrfs. I started using it that way too. It was nice to be able to take instant snapshots of my fs. I just amended my existing backup scripts to act on temporary snapshots instead. Then, I started dividing things up into subvolumes.

I suggest moving your filesystem root to its own subvolume (the convention is to call it @root), then optionally mounting the actual filesystem root as like /mnt/fsroot, and mounting it by passing subvol=/. That is where I keep my snapshots. You can still have /home be nested in @root. I have subvolumes for /var/cache, /var/lib/docker, etc. And more fragmented qemu images just go on a NoCOW subvolume with +C set, and I back up normally.

I also highly suggest reflinks. Once you start using them you will use them all the time.

I don't know if zfs supports reflinks yet, at the time it didn't. But combining reflinks with subvolumes solves a lot of problems. Since btrfs doesn't support recursive snapshotting, you can just do cp --reflink=always -rp @root /path/to/new-subvolume, and it will combine the contents of the subvolumes without consuming any more space. And of course, btrfs-send will (usually) not duplicate reflink data.

I did keep snapshots pruned to ~300 per subvolume.

But I also had around twenty subvolume I needed to snapshot in this manner.

N=1, though I suspect my case is going to sound a lot like the other complaints you might find. I thought btrfs looked great so I used it for the root and home filesystems on a laptop running OpenSUSE. (This is modestly significant, because SUSE specifically supports BTRFS and I figured it would give me a better experience.) Now the home filesystem never gave me any problems. The root filesystem, however, broke to the degree that I ended up reinstalling. Twice. I couldn't tell you the exact error anymore, unfortunately, but it looked like running out of space left the fs in an unrecoverable state.

Recovery from out of space errors is difficult on any copy-on-write filesystem since you need to write more data to disk to actually delete anything. You'll find similar stories about ZFS if you care to look for them. btrfs has had some protection against out-of-space issues for some time now (look for "global reserve" in `btrfs filesystem usage`). I think ZFS has something similar, but I don't care enough to check since I'm very careful to never let it get that far.

For my 2c, I haven't touch BTRFS since using it as the primary storage in a main storage server at work. We had a 6TB system reporting 3TB free (back when this was a lot of storage), yet every write operation (including deletes) was failing with an error indicating the disk was full. This was years ago, so I'm a little fuzzy on the details, but after some really painful downtime and troubleshooting, it turns out that BTRFS marks blocks as either metadata or data blocks. Once a block is marked, it won't be unmarked even if empty. So because this was a high churn system all of the blocks ended up marked as data and there was none available when more metadata was needed. The solution was running `balance` with some specific flags.

My understanding is that this is partially resolved now (empty blocks are automatically freed now), but not completely (partially empty blocks are never compacted without a balance, so you can still have a bunch of half-empty data blocks and no room for metadata), so one is a lot less likely to run into similar issues, but it was scarring enough to me that I've never run a BTRFS system again.

It's been a couple years since I last tried but I had file system corruption and some random nuisance issues that have long since slipped my mind. I never really got to the bottom of it, I had other things to do than fight my file system to get it working properly.

I tried precisely that, and had it corrupt my data… twice.

Bcachefs isn’t ready for prime time. I’m reasonably sure it’s better designed than Btrfs, and it’ll get there, but don’t do it yet. If Ubuntu becomes a problem, NixOS has 100% support for ZFS.

Were you able to report this? If you can get me enough information (e.g. via bcachefs dump), I'd like to track this down.

We talked on IRC. :-)

I'm planning to do that, but it'll have to be after my vacation; don't want to be running bcachefs on a remote server I can't fix.

For those following along, this sort of response is why I'm optimistic overall. And bcachefs was dramatically higher performance than ZFS, although the lack of an ARC is a particular pain point.

===

It's going to be an interesting bug, I'm pretty sure. It affected two sets of files: The Steam Linux runtime (Solder), and everything in /nix/store, but absolutely nothing else. Actually, I think it's probably a bug in the fsck routine; everything worked perfectly until I rebooted.

> And bcachefs was dramatically higher performance than ZFS

Would you care to go into some specifics?

On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio.

My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.

2GB/s is pretty low performance for storage these days. =-)

Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs.

When I write data to the setup, bcachefs and ZFS had very different behaviour.

ZFS:

- Writes to the HDDs first.

- Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.)

- Is in any case limited by how much can be buffered in RAM.

- Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC.

BcacheFS:

- Writes to the SSD first.

- Writes to the HDDs at its leisure.

- Can be told the NVMe is more reliable than the HDDs.

- Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue).

- Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up.

The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.

I don't think it's fixed yet?

biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet

I thought we might have :)

If it's a bug in fsck that will be fortunate, that'll be easy to track down.

I think it could be a very good replacement. In my opinion ZFS is still very static and disk/block based with some traditional RAID modes on top.

It can provide the same redundancy and safety but with tons more flexibility. For example having different redundancy levels using the same pool of disks and changing redundancy levels on the fly. It does feel like a huge step in in terms of "here are my disks" and then "please store this data this way". bcachefs then just does it in a reasonable and efficient way. Way better than having to set up all of your disk RAID configs upfront.

That being said it definitely still needs stability work. I tried it and it corrupted itself. So I love the idea, the design seems good overall. But it is going to need lots of testing and tweaking to become reliable. However it does seem like a solid addition to the filesystem space and I think being upstream will allow more people to work on it, avoid extra maintenance burden by staying in sync with changes and get the attention it deserves.

I wrote more about why I like the design and how it corrupted for me here: https://kevincox.ca/2023/06/10/bcachefs-attempt/

> My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time.

Wait. What? You'll forgive me, I have no experience with ZFS: why??

I have a fair amount of experience with ZFS, and have the same question.

Metadata scanning, I presume. It's not quite 30 seconds, but mounting my 4-drive 12TB btrfs volume (spinning rust) also feels like it takes upwards of ten seconds (I've never bothered to actually measure the mount time, so can't say with certainty).

Btrfs mounting of larger arrays became near-instant for me with the new block group tree enabled; see btrfstune: https://btrfs.readthedocs.io/en/latest/btrfstune.html

Importing ZFS pools from spinning disks can be slow. I'm guessing there's some serious random access going on because it's nearly instantaneous on SSDs

When importing/opening a ZFS pool, ZFS traverses the last few transaction groups written to the ZFS pool as well as the intent logs as a verification step to make sure that the ZFS pool is in a safe/consistent state before continuing (which could otherwise corrupt a ZFS pool even more and make the pool unrecoverable if something were to be wrong).

I'm not sure if this is the main bottleneck when opening a ZFS pool during boot, but I wouldn't be surprised if it were.

And if it is, personally I'm willing to sacrifice some boot time for some assurances that my data will be safer, but of course, not everyone's priorities are the same as mine :)

(Also, for the record, for me the pool import/opening process is nearly instantaneous on SSDs/NVMe and takes 2.4 seconds on a 12-year-old Intel Atom machine with a single HDD pool).

ZFS on Linux will not goes away. It is just that Ubuntu dropping it from the installer. You can still install it: https://launchpad.net/ubuntu/+source/zfs-linux/

And you can install it on a lot of distro too: https://openzfs.github.io/openzfs-docs/Getting%20Started/ind...

My current favorite is to install kABI-tracking kmod on RHEL. (But their kABI-tracking is not a guarantee as they uses more.) In fact I move away from Ubuntu knowing that they knowingly ship a ZFS version with file corruption bug, which is introduced by their own patch of OpenZFS.

So if OpenZFS is serving you well, don't worry about the Ubuntu situation because there's plenty better options.

Reply on news.ycombinator.com