> Setup alerting if you care about your data
Yes. and make sure that you have the correct signal to noise ratio. Otherwise you won't act on it.
> scrub everything
Yes, this too.
One thing that I would add is avoid mdadm like the plague. Its tricky, poorly documented and requires a whole bunch of other crap tools to make it work.
ZFS or proper hardware raid, there is no inbetween.
On the subject of loosing a raid during rebuild, I've had that, but I can't be sure that it was because of simultaneous disk failures. The raid in question was a 60 drive array with 8TB drives. There were 4 raid 7 groups with 4 spare drivers.
After each rebuild, a new drive would fail. over the space of two weeks we changed about 6 drives. Because the raid was under huge pressure, it didn't have the time needed to rebuild (24 hour rebuild time jumps up to 96 hours)
In the end we migrated the data to another array.