Causing ZFS corruption for fun, profit, and quality assurance

datto.engineering

113 points·binwiederhier··6 comments

ZFS volumes can also be backed by files. In this case, the volumes can be corrupted by opening the backing file in a text editor, and perform a search/replace. This is simpler but of course less interesting :-)

Another interesting article I've mentioned in another post, is a demonstration of ZFS' self-healing capabilities: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.

Being able to back volumes by files is an amazing feature for testing. I've lost count of the number of times I've done this to prove some documentation works before applying it to the actual storage array in question.

Some device-mapper virtual block device types also provide ways to simulate error conditions. For bad block errors there is dm-dust.

https://www.kernel.org/doc/Documentation/device-mapper/

This was an interesting read! I hope they do a follow up article on how they automatically manage the rest of their recovery process.

If with "recovery", you intend recovering that file, it's not possible, since all the copies available in the mirror are corrupted.

In this case, the recovery (repair) of the system (that is, to bring it to a clean state) simply consists in deleting the file (see https://docs.oracle.com/cd/E26505_01/html/E37384/gbbbc.html#...).

If there was an intact copy of the file available in the mirror, then one doesn't need to do anything, as ZFS is self healing: https://blogs.oracle.com/timc/demonstrating-zfs-self-healing.

Presumably they're referring to Datto's recovery process (i.e. recovering automatically from secondary or tertiary backups), as opposed to just the ZFS mirror.

Reply on news.ycombinator.com