Thanks for the reply – it sounds like you're definitely hitting the areas I care about.

One quick thought: I was at USENIX the year that ZFS was announced and one thing which was very popular was when the Sun team talked about how much random testing they'd done with billions of runs exercising the major features with simulated failures, memory corruption, etc. so they could stress-test the detection and recovery mechanisms:

https://blogs.oracle.com/bill/entry/zfs_and_the_all_singing

You could see a lot of sysadmins nodding their heads as they described each class of failure mode and how they were trying to simulate it.

These days I'd probably toss in something like running afl-fuzz against your storage code, too.