Hacker News Reader

Comment by wtallis | Hacker News Reader

> "Here's a fact: the upstream's official statement is that RAID 1 isn't production-ready."

That doesn't seem to match anything I've seen. The btrfs status page classifies RAID1 and RAID10 features as "mostly OK". The one and only documented caveat is that when a disk in a RAID1 fails, you should mount the filesystem as read-only until you are ready to fix the problem by either replacing the failed disk or converting to a non-RAID profile. There's a real bug underlying this limitation, but the only way to encounter this bug is to be quite cavalier about how you handle a degraded array, by doing something that you shouldn't expect to be safe.

View on HN

Mostly OK != production ready.

A degraded array should have no substantially greater risk of failure than the single disks the vast majority of people rely on daily. I can think of one obvious reason one might mount the degraded array simply to copy data from it to a different arrangement. Another reason is in a non enterprise use case a desktop users might not instantly have a disk available and might mistakenly believe that this is no more risky than having only one disk in the first place.

This is ridiculously flaky for something that is supposed to be years in development with backing from major companies.

This confirms my feeling that btrfs belongs in the waste basket.

> Mostly OK != production ready.

Don't nitpick about what a two-word summary implies when there's only one underlying bug at issue. Just address the bug itself. "Production ready" never means 100% bug-free.

> A degraded array should have no substantially greater risk of failure than the single disks the vast majority of people rely on daily.

If you are comfortable dropping from the RAID1 profile to a non-RAID profile, btrfs requires you to explicitly make this conversion; there's no safe way to make it automatic. Forcing the filesystem to accept writes while it's still in RAID1 mode but is incapable of providing RAID1 data integrity is something that you should expect to cause problems.

> I can think of one obvious reason one might mount the degraded array simply to copy data from it to a different arrangement.

You can mount read-only as many times as you want if you're going to copy the remaining data to a different filesystem. If you want to do an in-place recovery, the current limitation is only that you shouldn't mount the degraded array as writable until you're ready to make it not degraded anymore.

Ran into this with my RAID1: https://www.spinics.net/lists/linux-btrfs/msg62346.html. I then ditched btrfs, and I'll never try it out again.

> I wish that if a drive fail, the btrfs filesystem still mounts rw and leave the OS running, but warns the user of a failing disk and easily allow the addition of a new drive to reintroduce redundancy.

This might make sense for you, but is insane as a default policy. Manual intervention should be required before the FS will accept writes with less than the configured degree of redundancy. Silently mounting and hoping the user notices something in their logs is too dangerous.

> I created a raid1 btrfs filesystem by converting an existing single btrfs instance into a degraded raid1, then added the other drive

This seems backwards. Why not add the second drive, then convert to RAID1?

Note that the patch to work around the refusal to mount is extremely simple, and that the patch is quite safe if used properly. But it's not really an acceptable solution for upstreaming, because it will lead to bigger problems if used in more complicated situations. There are several potential solutions that would be safe and widely deployable, but all involve changing far more than two lines of code.

Agreed, that default policy should be manual intervention, but currently I don't see how I can set a different policy.

For example in a large cluster where that degradation is expected and monitored for.

How do you add a second drive? Is there a brtfs raid faq/howto?

In a large cluster, you'd probably plan on replacing every drive that failed rather than reconfigure to use less redundancy. So in that case, you'd probably want the hot spare feature to be stabilized and upstreamed. Then the FS could automatically copy over (or reconstruct from parity) data for a missing disk, without modifying data on the surviving disks.

In environments with a smaller budget that want to get the system back up and running before a replacement drive is available, it could be valuable to be able to pre-specify that the system should rebalance with less redundancy when a drive goes missing. I'm not aware of any work to implement this kind of feature. No enterprise customer would want or use this feature, and even a home user on a shoestring budget wouldn't necessarily want this rebalancing to happen automatically. (What if the drive was only temporarily missing, such as from a failed or loose SATA cable? You wouldn't want to do a ton of re-writing of data only to have to reverse it on the next boot.)

A hot spare is expensive. I want pure redundancy. That is making a one drive failure into a still perfectly operational server/node/box.

That's what RAID1 usually means.

And sure, you can't survive the next without replacing the failed drive and resyncing.

But the remount read only thing is something different. It's a useful failure mode, but doesn't help with operational simplicity.

(If the SATA cable is loose, the it'll cause intermittent failures, you'll see it in the log, and there will be a lot of resync events. And probably degraded performance, a lot of ATA (or SCSI errors in case of SAS), and other bus/command errors that go away on retry. And with SMART it's possible to at least guess that it's not the drive. It'd be great to have an error notification interface from the kernel and a tool could try to dig into the relevant subsystem's perf and health data to try to guess what's the faulty component exactly.)

> but the only way to encounter this bug is to be quite cavalier about how you handle a degraded array, by doing something that you shouldn't expect to be safe.

The way to encounter it is to mount the array a second time. That's hardly "cavalier handling".

A decade in, and RAID 1 doesn't work right. Saying this clearly offends the feels here at HN, but it's a fact.

> The way to encounter it is to mount the array a second time. That's hardly "cavalier handling".

`mount` is not the same as `mount -o degraded,rw`. The latter should raise the eyebrows of anyone paying attention to what they're typing. Odds are that you'll have to consult the docs to even find these mount options, because it doesn't happen automatically. This is where any careful, sane user who's concerned about their data would spend a few more minutes thinking through their entire recovery procedure.

There's one corner case to RAID1 recovery where the current tooling does not fully prevent a careless user from putting the FS in a bad state. This is not the same as "RAID1 doesn't work right".

I don't think a careful, sane user who's concerned about their data would be on btrfs in the first place.

No, probably not. But this particular bug isn't the reason why users who need very high level assurances should avoid btrfs for now. The nature of this bug does not does lend itself to using it as an argument that btrfs is unsafe in general.

Consider that most users will be careless, because running a filesystem to them is as exiting as any of the other dozen OS services. Having a corner case like this in RAID1 to me is like having a corner case in encryption... That I immediately run into this case on my first btrfs RAID1 problem, makes me think that there is a steep slope downwards to this corner.

If when a drive in your RAID fails, your first response is anything but checking the docs for the recovery procedure, you're going to end up disappointed sooner or later. When your fault-tolerant system tells you something broke leaving it in a fragile state and your response is to tell the system to ignore that and pretend everything is normal, expect trouble. You're working in a domain where there is no one right answer, so the system cannot magically anticipate how to handle the exceptional situation. None of the above is the fault of btrfs. None of this is avoidable. This problem has to be faced by ZFS, too.

What is avoidable is that btrfs could make it harder to do the equivalent of `mount -o degraded,rw`. But ultimately, there will be some mechanism for modifying a degraded array, and it'll get documented and then excerpted in blog posts and StackOverflow answers without all the context, and users will find a way to work themselves into a corner. There are all kinds of ways to do this with ZFS, too. ZFS tends to default to the approach of requiring you to copy all your data elsewhere and rebuild your array from scratch. What btrfs is doing here is no worse, except that it's a bit less up-front about the limitation because it's actually completely avoidable and this is a fixable UI bug, not a deep-seated architectural limitation.

Do you have a link to this real bug ? It would be interesting to follow it until it is resolved.

There are plenty of instances of this bug being brought up on the mailing list. One of them is already linked elsewhere in this discussion, and the btrfs status page (also linked from this discussion) has further mailing list links.

Basically, btrfs doesn't want to allow a writeable mount when it might be missing some data. If there's some data on the FS that isn't stored with the RAID1 profile, then the kernel can't safely assume that the missing drive didn't have more chunks like that, holding data that wasn't mirrored on one of the surviving drives. But it's currently not possible to convert from RAID1 to non-RAID or to rebuild the array with a replacement without mounting the degraded array as writeable, which leads to non-RAID data being written. That puts the FS in a state that cannot be automatically judged safe at mount time, and the FS remains in that state until the recovery is complete (either converting from RAID1 to non-RAID, or replacing the failed drive).

There's no easy way to require the user to specify at the time of the `mount -o degraded,rw` whether they intend to resolve the situation by ceasing to use RAID1 or by replacing the failed drive. That leaves users with the opportunity to do neither and instead make the situation worse.

Thanks for the explanation. I was hoping for a Github issue number (or Bugzilla or whatever) to easily track this bug, but perhaps the Btrfs dev team doesn't work with issue number ?

At least for RAID1, it seems that implementing RAID1 N-way mirroring would ease the process to recover from a failed drive. In case of drive failure, we could use the remaining drive in read-only mode to copy the data to a new drive, hence creating a RAID1 array with two working drives and one failed drive. The OS should then allow to boot in rw mode, and from there it is easy to remove the failed drive from the RAID1 array.

However it seems that RAID1 N-way mirroring (with N > 2) is not even on the roadmap at this moment.

Have I misunderstood something or does this approach make sense ?

You can do RAID1 with more than two drives, but you'll only get two copies of each chunk of data. In this scenario, when one drive dies you can still write new data in RAID1 to the remaining space on the surviving drives, so mounting the FS writeable in degraded mode doesn't risk leaving the FS in a state where the safety is hard to determine on the next mount. If space permits, you can also rebalance before even shutting down to remove the failed drive, also avoiding the corner case.

Being able to do N-way mirroring with three or more copies of the data would be nice, but it's not necessary; 2-way mirroring across 3 or more drives is sufficient, and the hot spare feature will be more widely useful.

I'm not sure that I have understood.

I was referring to this sequence of events: 1) 2-way mirroring across 2 drives 2) one drive fails 3) buy and plug a new drive 4) rebalance to have 3-way mirroring across 3 drives (with one being out): this is currently not possible 5) remove the failed drive, ending with 2-way mirroring across 2 drives

But it seems that you are referring to: 1) 2-way mirroring across 3 drives 2) one drive fails 3) rebalance to have 2-way mirroring across the 2 working drives 4) remove the failed drive, ending with 2-way mirroring across 2 drives

I assume that people don't/won't start the initial RAID1 with 3 drives.

Anyway, I would find 3-way mirroring across 3 drives very useful as it gives a simple identical foolproof process to replace a faulty hard drive, whether it has just a few corrupted data (but still readable) or have completely failed : just plug a new drive, rebalance, reboot and remove the defective drive.

> rebalance to have 3-way mirroring across 3 drives (with one being out): this is currently not possible

I'm not sure this even has meaning. But anyways, it's probably pointless to try to kick off a rebalance when the FS is still trying to use a dead drive. Either use the device replace command (which isn't stable yet), or tell btrfs to delete the dead drive then add the replacement drive. If the problem drive is failing but not completely dead yet, then the device replace command is supposed to move data over with a minimum of excess changes to drives other than the ones being removed and added. But the device replace command doesn't properly handle drives with bad sectors yet, so the separate remove and add actions are more reliable albeit slower and put more work on the other drives in the array.

You are probably right, bad idea.

Where can I find the proper official procedure to replace a failed drive (i.e. which cannot be mounted anymore) in a RAID1 array (with 2 drives) ?

I found these 2 links: https://unix.stackexchange.com/questions/334228/btrfs-raid1-... https://unix.stackexchange.com/questions/227560/how-to-repla...

But it is written there that if 'replace' doesn't work, it can take up to 5 days using add/remove for only 100GB !

I haven't found any official procedure on the Btrfs wiki.

Reply on news.ycombinator.com