Comment by kevincox | Hacker News Reader

Comment by kevincox | Hacker News Reader

Parent Full thread

IMHO bcachefs has important advantages over ZFS. It is far more flexible. ZFS is really similar to traditional block-based RAID. You can get pretty flexible configurations but 1. They are largely fixed after creation and 2. They only operate at "dataset" level granularity.

bachefs has a really flexible design here where you basically add all of your disks to the storage pool and then you can pick redundancy and performance settings per folder (arbitrary subtrees, not just datasets decided at setup time) or even file. For example you can configure a default of 2 replicas for all data, but for your cache directory set it to 1 replica. If you have an important documents folder you can set that to 3 replicas, or 4.2 erasure coding.

Similarly you can tell it to put your cache folder on devices labeled "ssd" but your documents folder should write to "ssd" but then be migrated to "hdd" when they are cold.

And again, all of this can be set at any time on any subtree. Not just when you initially set up your disks or create the directories.

If you mix a small SSD and a large HDD into a single bcachefs pool, and set it to 2 replicas, do writes have to succeed on both devices before returning success? I.e. is performance constrained by the slowest device? And what happens when the small SSD fills up? Does it carry on writing to the HDD but with only one replica, or does it report no space left?

You explicitly configure the cache, as laid out here: https://bcachefs.org/Caching/

In the common case that you mentioned, data present on the full SSD would be overwritten "in standard LRU fashion"; meaning the "Least Recently Used" data would no longer be cached. New data would be written to the SSD while a background "rebalance thread" would copy that data to the HDD. I assume that the "sync" command would wait for the "rebalance thread" to finish, though I will admit my own ignorance on that front.

Are you trying to ask if bcachefs lets you set up stupid configurations? If you only have two devices and you configure it use replication to store two copies of all your data, then you are unavoidably constrained by the capacity of the smaller device. Whether the smaller device is a hard drive or SSD is irrelevant, because neither copy can be regarded as a discardable cache when they're both necessary to maintain the requested level of redundancy.

I am trying to understand bcachefs by asking about an edge case that might be illustrative.

> you are unavoidably constrained by the capacity of the smaller device

Sure, so what does bcachefs actually do about it? ENOSPC?

Answers to the question on synchronous write behavior also welcome.

> I am trying to understand bcachefs by asking about an edge case that might be illustrative.

It looks more like you're questioning whether the filesystem that Linus just merged does obviously wrong things for the simplest test cases of its headline features. Has something given you cause to suspect that this filesystem is so deeply and thoroughly flawed? Because this doesn't quite look like trying to learn, it looks like trying to find an excuse to dismiss bcachefs before even reading any of the docs. Asking if everything about the filesystem is a lie seems really odd.

> Asking if everything about the filesystem is a lie seems really odd.

This person was asking pertinent questions, no need to bash it.

I lost some data with XFS after it was declared "stable" in the linux kernel. Every day at around 00:00 the power will fail for around 2 seconds. Other filesystems (ext2, jfs, reiser) will do a fsck, but xfs was smarter. After 2 or 3 crashes the xfs volume will not be usable anymore (no fsck possible).

So yes, some of us do need more than a "trust me, it's ok" when we are talking about our data.

The #1 point of ZFS is protections again bitrot, ie. checksums on all data. Does bcachefs do this?

https://bcachefs.org/

It's literally the second item listed on the main web site

Full data and metadata checksumming

Thanks, that's good news. Unfortunately it seems bcachefs isn't using pools like ZFS but instead, like btrfs, creates file systems directly on a collection of disks. That's a bummer if true.

What are the use cases where you find micromanaging vdevs worthwhile compared to automatic storage allocation that can be changed and rebalanced later? bcachefs does offer several per-device controls over data placement and redundancy that btrfs lacks; are those still not enough?

Exactly, I _don't_ want to micromanage devices so I throw them all into a pool (with defined redundancy) and then I allocate file systems and vdevs (for iSCSI) from that. With btrfs (and bcachefs) I have to manually assign devices to file systems and I can't have multiple file systems (well, you can have sub-volumes but you can't avoid a big file system in that collection of devices).

Maybe I'm missing something, but the pool abstraction makes this very clean and clear.

> so I throw them all into a pool (with defined redundancy) and then I allocate file systems and vdevs (for iSCSI) from that.

That sounds backwards. Don't you have to manually define the layout of the vdevs first in order to establish the redundancy, and then allocate the volumes you use for filesystems or iSCSI? If you just do a `zpool create` and give it a dozen disks and ask for raidz2, you're just creating a single vdev that's a RAID6 over all the drives. There's an extra step compared to the btrfs workflow, but if you're not using that opportunity to micromanage your array layout I don't see why you'd prefer that extra step to exist.

> and I can't have multiple file systems (well, you can have sub-volumes but you can't avoid a big file system in that collection of devices).

Isn't this a purely cosmetic complaint? With at least btrfs, you don't even have to mount the root volume, you can simply mount the subvolumes directly wherever you want them and pretty much ignore the existence of the root volume except when provisioning more subvolumes from the root. You can pretend that you do have a ZFS-style pool abstraction, but one that's navigable like a filesystem in the Unix tradition instead of requiring non-standard tooling to inspect.

If you wan't volume manager features, you can put btrfs/bcachefs on top of LVM

bcachefs currently has full metadata and data checksumming, but there is no scrub implementation. This will likely be in the works soon now that it has been merged.

that actually sounds quite neat

Reply on news.ycombinator.com