Didn't sequential resilver already solve the read half of the problem? Because that would already cut it down from weeks.
The point with DRAID is that the spare capacity is distributed across all the disks in the vdev, hence the name. Thus all disks are read from and written to when "replacing" a failed disk with the spare capacity.
The combined throughput of the multiple disks is typically much higher than that of the single disk.
In addition they have an additional constraint which allows DRAID to also support sequential resilver.
At least that's my understanding.
There's been several presentations of DRAID on the OpenZFS channel, but the latest one is here[1]. There's also a writeup here[2], though not sure if it's 100% up to date, though should be close enough for getting the concept.
[1]: https://www.youtube.com/watch?v=jdXOtEF6Fh0
[2]: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAI...
Additional speed on top of that is nice of course, if you're using such wide pools.
Not sure how much that matters in practice, I've just got a home NAS with 6 disks to play with.
Is it the fact that you get 12tb of essentially non sequencial io mixed with random reads during the rebuild? Still 20 mb/s seems 1990s slow writing tio a mechanical HD?
Ed: ok, ok, from tfa:
> For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds can take weeks. Resilver I/O activity is deprioritized when the system has not been idle for a minimum period. Full zpools get fragmented and require additional I/O’s to recalculate data during reslivering.
Obviously writing 12 tb at 0 mb/s will take time..
If you insert a cold spare to replace a disk — yes, you have to write the full disk.
If you migrate a disk from a hot spare to an active disk, it’s already largely populated, so you only have to write the missing chunk.
At least, that’s my limited understanding/reading. I’m not sure I fully grok what is happening as there aren’t necessarily dedicated spares in this setup.
But I think the difference in your scenario is hot vs cold spares.
When you do a rebuild with a real, distinct physical disk, there's no special benefit, you're still almost certainly going to bottleneck on the write speed of that single disk.
With raid 6 for example, I pop in a new drive, and all the write/repair hits only that drive.
With a virtual spare, all drives end up with massive writes, meaning I might hit write errors, thus borking a raid which would have been ok for reads.
I realise there are 100 'what ifs' and scenarios here, but when my raids tend to drop a drive, it's on heavy write ops, not reads. I'd say 90% of the time.
I imagine ssd cascading/eol write failures might make this virtual spare failure scenario even more likely.
(yes I am aware of sector reallocation delays on spinning disks, I am referring to real EOL scenarios on drives during writes, being far more common for me than reads)
I suppose it works out as follows. With traditional raid, you are rebuilding while degraded from a safety perspective. So obviously, if your configuration guarantees losing `n` drives while healthy does not cause data loss, you could lose data during rebuild if `n` drives failed.
With the distributed hot spare approach, you are no longer degraded from a safety perspective as soon as the distributed spare's data is written. This is faster than traditional raid, as the writes are distributed among many disks. So time-wise you get out of data safety degraded state faster. (It does seem like it might be riskier though, since you are doing heavy writes to many drives while safety degraded, instead of only writes to the spare).
So now you replace the failed drive, and the array rebuilds to make this new spare's capacity distributed. This will take a long time, since you will be writing to basically this whole drive as part of distributing its spare space. (Plus probably shuffling chunks around between many other drives). This could possibly take even longer than traditional raid rebuild. However during this whole process, your array is not in degraded state from a data safety point of view.
Or to summarize: during the long single-drive-write-limited part of replacing the failed drive with a new hot spare, you are not degraded from a safety of data point of view.
Yes. With the caveat that you'll be *reading* from all of the disks to satisfy those writes. In a traditional setup, you'd have multiple striped raidz{,2,3} vdevs. When you swap in a spare, you're only pulling reads from the disks in the same vdev. This will drive the performance from this vdev to the ground. In this draidz setup, you'll be pulling from all of the disks, so the entire pool is stressed at the same rate.
I think the way I'm starting to think about it is that each drive has X amount of spare capacity or buckets. But they are otherwise used. When a drive goes out, the primary data from that drive is distributed across the array to the already-in-place spare capacity. So, yes, after the primary data is mirrored out, the pool is no longer degraded. But, I think the point being made here is that you'll still want to replace the drive, which will still cause *that* drive to be fully written.
I don't think from a data safety or performance perspective pre dRAID vdev really differ too terribly much from raid 50 setups. (They do differ greatly in administrative flexibility, versus the insanely rigid approachs of traditional hardware raid).
But yeah, after replacing the failed drive, ideally dRAID would basically do just what it did after drive failure in reverse. (End result: No longer having any drives holding data for a different vdev than they belong to). Obviously this means writing basically that whole disk, and reading from potentially every disk. But as you mention you are spreading the load out across the whole array, hopefully reducing the overall impact, and being less likely to stress any drive into dying.
Let's say you've got an array with 10 disks and 100TB of data. You experience a single-disk failure.
If you're running a RaidZ, you have to write the lost data (100TB / 10 disks) to the single replacement disk. Writing 10TB of data to a single disk takes quite a while, especially on inexpensive SATA disks.
If you were running a dRAID with equivalent redundancy settings, you have to write (100 / 10)TB to the 9 remaining disks in the array. Each disk will have to write about 1.2 TB of data in parallel.
Writing 1.2TB to each disk should be roughly ~5-10x faster than writing 10TB sequentially.