Hacker News Reader

Comment by walrus01 | Hacker News Reader

I am not sure why this person feels compelled to use btrfs raid1 when mdadm raid1 exists and works just fine at the block device level.

View on HN

BTRFS RAID1 has numerous advantages compared to regular RAID1. It does not mirrors whole disks, but chunks of the disk, usually 1GB chunks. This has numerous implications: - drive size can be mixed without usable space loss (kind of, with some combinations you end up losing a bit) - you don't need an even number of drives - you don't need a raid0 atop several raid1 to use e.g. 6 drives, it's all one big raid - a drive can be swapped for a bigger or smaller drive without issue, meaning spare drives don't need to be identical

etc.

Just use LVM thin-provisioned RAID1 volume.

This is such a great feature, it makes it really easy and convenient to expand btrfs arrays. It's such a pity that the parity support is still unstable. Now that ZFS has RAID-Z expansion I don't need to wait anymore.

How would a Btrfs RAID1 array of (equally sized) spinning rust and SSDs work? Will it slow down to the speed of the HDD, or will I get SSD speeds and will the HDD eventually catch up? If not by default, then by option?

For the reading, btrfs currently chooses one of the drives at random, I think based on the PID of the process that's doing the reading.

For writing, you have to wait for the slow one.

If you want to mix SSD and hard drives then you probably want bcache.

If a bit is flipped on one of the SSD, btrfs can recover using checksum.

Not sure how useful it is, but this is what btrfs/zfs people say

Happened on my NAS: had a corruption due to a bitflip, BTRFS recovered fine.

As a ZFS user: quite useful. I’ve had numerous of cases of corruption repaired by routine scrubs. Almost every other filesystem (btrfs being an exception afaik) would have been none the wiser.

Useful for HDD, but on SSD? When both the NAND and Controller has Error Correction?

It's just as useful on either. Error correction predates SSDs.

Errors can happen during transmission as well, especially if cables or backplanes are involved.

In addition to cables and backplanes that have been mentioned, the entire path from RAM to PCIe to disk controller has non zero risks of errors. That said I'm pretty sure that the BTRFS errors I've seen from SSDs have all been the fault of the SSDs in question.

Also note that hard drives have had internal error detection and correction codes for decades, so SSDs doesn't change things much. Probably most checksum errors are misplaced writes and missing writes.

HDDs also use ECC hence why read errors occur on failing drives instead of just silently returning corrupt data. That's not an ironclad guarantee and plenty of places in the data path can let an errant cosmic ray flip bits. BTRFS doesn't even eliminate this, it just reduces the surface area and most importantly gives you a way to durably detect or correct this from when the checksum is generated in the kernel onwards.

+1, having the visibility (and the ability to correct it) is also often a leading indicator of drive failure.

I would argue though that at this point most other filesystems have this, but they're not as ergonomic/obviously built in. LVM has integrity checking at the volume level (if you give drives to the pool instead of giving the pool an already-RAIDed device), mdraid has dm-integrity. The options are there but not quite as discoverable.

In the post I wrote 'between the 2 SSDs SMART reports 18 uncorrectable errors and “btrfs device stats” reports 55 errors on one of them'. That's 37 errors that BTRFS found that mdadm wouldn't have found. 37 errors out of 500G of data isn't a lot, but it depends where it occurs, if it's in the root directory then it will mess up your day (annoyingly BTRFS doesn't track where the errors were so you can't easily determine what the implications of not having BTRFS might be). That is not unusual, I have had similar things on almost all my systems that run BTRFS or ZFS.

Also I use snapshots too.

Exactly, especially if he is using snapshots, BTRFS can cause serious write amplification for frequently snapshotted and changed data.

I'm not sure it makes a lot of difference unless you have hundreds of snapshots. The way changes work to BTRFS is that every change without snapshots does most of the same stuff that a change with snapshots does. If however you have many snapshots (more than 500 according to rumours, I've seen it with over 1000) performance can get really bad. If your script to make snapshot backups works and the script to remove old ones doesn't work properly then the system runs fine for months and suddenly goes ridiculously slow.

A VM with a BTRFS filesystem running on a host using BTRFS gives serious write amplification. But if the real storage is SSD then it's not too bad.

What's nice is that as of Linux 5.16 BTRFS now supports ZNS (https://zonedstorage.io/) to drastically reduce write amplification compared to traditional SSDs by bypassing a lot of the FTL and moving to an interface that's similar to host managed SMR drives.

Reply on news.ycombinator.com