Hacker News Reader

Out of curiosity, why weren't they running the metadata drive in a mirroring raid? If you have PB of data, wouldn't it make sense to spend the ~$100 for a second 3TB drive to mirror your metadata?

Or was the inode problem not a local disk problem but a problem in the Luster fs? I couldn't quite tell from the article.

View on HN

It's almost a certainty that the MDS (metadata server) was situated on a mirrored RAID (prob RAID10). I'm guessing that the RAID system itself (software MDRAID or some HW array, DDN or something like a NetApp E-Series) corrupted the data under the FS that the MDS used, which I'm also assuming was XFS.

Lustre, for those who don't know it, is a cluster meta-filesystem, with separate metadata and object servers, each sitting on top of host file systems/RAID/storage.

The metadata target (MDT) in the MDS is actually "ldiskfs" which is an enhanced version of ext4. One possibility may be to use ZFS in the future as the support in Lustre seems to be quite stable now.

It seems pretty impossible to find out the exact root cause in retrospect as the system was running for a long time without apparent issue. Any ideas are welcome though.

It was filesystem-level corruption in Lustre. The underlying disk arrays and other hardware have comprehensive redundancy.

Reply on news.ycombinator.com