Hacker News Reader

Comment by magicalhippo | Hacker News Reader

When replacing a disk with a hotspare in a regular RAID-Z vdev, the process is limited to the sequential write speed of that single new disk.

The point with DRAID is that the spare capacity is distributed across all the disks in the vdev, hence the name. Thus all disks are read from and written to when "replacing" a failed disk with the spare capacity.

The combined throughput of the multiple disks is typically much higher than that of the single disk.

In addition they have an additional constraint which allows DRAID to also support sequential resilver.

At least that's my understanding.

There's been several presentations of DRAID on the OpenZFS channel, but the latest one is here[1]. There's also a writeup here[2], though not sure if it's 100% up to date, though should be close enough for getting the concept.

[1]: https://www.youtube.com/watch?v=jdXOtEF6Fh0

[2]: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAI...

View on HN

What I was questioning was that it shouldn't be taking weeks because the write speed of a single disk is enough to fill it in a day or two with sequential IO. I.e. the sequential resilver alone should be enough to fix the "can take weeks" problem.

Additional speed on top of that is nice of course, if you're using such wide pools.

With DRAID you'd cut down the time to an hour or so though.

Not sure how much that matters in practice, I've just got a home NAS with 6 disks to play with.

I get that using hot virtual spares is much better, I also wonder at a week to write 12tb. 12tb at 200 mb/s is roughly 20 hours.

Is it the fact that you get 12tb of essentially non sequencial io mixed with random reads during the rebuild? Still 20 mb/s seems 1990s slow writing tio a mechanical HD?

Ed: ok, ok, from tfa:

> For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds can take weeks. Resilver I/O activity is deprioritized when the system has not been idle for a minimum period. Full zpools get fragmented and require additional I/O’s to recalculate data during reslivering.

Obviously writing 12 tb at 0 mb/s will take time..

That kinda makes no sense though, because the whole replacement disk needs to be written anyway. Presumably before the failure you were using some part of the old disk, so you need to fill up the new disk to the same level before you're in the same state.

I’m thinking of this in terms of hot spares, not cold spares.

If you insert a cold spare to replace a disk — yes, you have to write the full disk.

If you migrate a disk from a hot spare to an active disk, it’s already largely populated, so you only have to write the missing chunk.

At least, that’s my limited understanding/reading. I’m not sure I fully grok what is happening as there aren’t necessarily dedicated spares in this setup.

But I think the difference in your scenario is hot vs cold spares.

The primary benefit of dRAID is when you're doing a replace with a virtual "distributed hot spare", where it uses spare capacity distributed across all the disks in the vdev to replace a failed disk, thereby letting the rebuild scale with the total number of disks in the vdev.

When you do a rebuild with a real, distinct physical disk, there's no special benefit, you're still almost certainly going to bottleneck on the write speed of that single disk.

Not sure I like this, for many spinning disk raids of that scale I have are mostly read, rarely write.

With raid 6 for example, I pop in a new drive, and all the write/repair hits only that drive.

With a virtual spare, all drives end up with massive writes, meaning I might hit write errors, thus borking a raid which would have been ok for reads.

I realise there are 100 'what ifs' and scenarios here, but when my raids tend to drop a drive, it's on heavy write ops, not reads. I'd say 90% of the time.

I imagine ssd cascading/eol write failures might make this virtual spare failure scenario even more likely.

(yes I am aware of sector reallocation delays on spinning disks, I am referring to real EOL scenarios on drives during writes, being far more common for me than reads)

While I can see the hot-spare advantage here, I do have to wonder about the impact of replacing the broken drive.

I suppose it works out as follows. With traditional raid, you are rebuilding while degraded from a safety perspective. So obviously, if your configuration guarantees losing `n` drives while healthy does not cause data loss, you could lose data during rebuild if `n` drives failed.

With the distributed hot spare approach, you are no longer degraded from a safety perspective as soon as the distributed spare's data is written. This is faster than traditional raid, as the writes are distributed among many disks. So time-wise you get out of data safety degraded state faster. (It does seem like it might be riskier though, since you are doing heavy writes to many drives while safety degraded, instead of only writes to the spare).

So now you replace the failed drive, and the array rebuilds to make this new spare's capacity distributed. This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives). This could possibly take even longer than traditional raid rebuild. However during this whole process, your array is not in degraded state from a data safety point of view.

Or to summarize: during the long single-drive-write-limited part of replacing the failed drive with a new hot spare, you are not degraded from a safety of data point of view.

> This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives)

Yes. With the caveat that you'll be *reading* from all of the disks to satisfy those writes. In a traditional setup, you'd have multiple striped raidz{,2,3} vdevs. When you swap in a spare, you're only pulling reads from the disks in the same vdev. This will drive the performance from this vdev to the ground. In this draidz setup, you'll be pulling from all of the disks, so the entire pool is stressed at the same rate.

I think the way I'm starting to think about it is that each drive has X amount of spare capacity or buckets. But they are otherwise used. When a drive goes out, the primary data from that drive is distributed across the array to the already-in-place spare capacity. So, yes, after the primary data is mirrored out, the pool is no longer degraded. But, I think the point being made here is that you'll still want to replace the drive, which will still cause *that* drive to be fully written.

Right, the free space buckets are a fine analogy. They are conceptually similar in some ways to the free space in the hot standby drive.

I don't think from a data safety or performance perspective pre dRAID vdev really differ too terribly much from raid 50 setups. (They do differ greatly in administrative flexibility, versus the insanely rigid approachs of traditional hardware raid).

But yeah, after replacing the failed drive, ideally dRAID would basically do just what it did after drive failure in reverse. (End result: No longer having any drives holding data for a different vdev than they belong to). Obviously this means writing basically that whole disk, and reading from potentially every disk. But as you mention you are spreading the load out across the whole array, hopefully reducing the overall impact, and being less likely to stress any drive into dying.

To clarify, you're redistributing the missing data across all of the other disks in the array simultaneously, rather than to one disk sequentially.

Let's say you've got an array with 10 disks and 100TB of data. You experience a single-disk failure.

If you're running a RaidZ, you have to write the lost data (100TB / 10 disks) to the single replacement disk. Writing 10TB of data to a single disk takes quite a while, especially on inexpensive SATA disks.

If you were running a dRAID with equivalent redundancy settings, you have to write (100 / 10)TB to the 9 remaining disks in the array. Each disk will have to write about 1.2 TB of data in parallel.

Writing 1.2TB to each disk should be roughly ~5-10x faster than writing 10TB sequentially.

Reply on news.ycombinator.com