Hacker News Reader

Comment by Filligree | Hacker News Reader

Without ECC RAM, you're far more likely to get uncorrectable / unnoticeable corruption.

This is not unique to ZFS, and it doesn't make ZFS worse than other filesystems. But since the reason you'd use ZFS is often to avoid any corruption, it's tradition to advise the use of ECC.

View on HN

I learned this the hard way on an old server that did not have ECC.

I had a file server happily ticking away using ext4.

Converted it to ZFS - and a week later got file system corruption reported. Ran a very extensive memory test - and sure enough I had bad RAM (but it took 2-3 days for the errors to show up).

In the wild there has to be a ton of corruption that just never gets discovered without end to end checking.

If you have a large jpgs or MKVs - a flipped bit here or there is not going to be apparent.

Why would ZFS be worse than ext4 or anything in this way if you don't have ECC?

Genuine question, I don't understand this claim. As far as I can see, ZFS provides protection against some types of failures on disk, which ext4 doesn't. ECC has no impact on that, it protects another dimension.

I read this as they've been having corruption happening all along, but only ZFS detected it?

Yes - exactly.

To be clear, ext4 did not cause the corruption - but neither did it detect it or correct it. It happily sent corrupt blocks out to disk.

I think this happens a lot more frequently than people realize.

One reason is that you generally perform scrubbing, thus you are potentially rewriting data which would otherwise be at rest. If your memory is bad, this could replace good data with bad. FS that doesn't scrub doesn't have this issue.

No it is worse with ZFS because it doesn't have a fsck tool. If you have a bit flip in the ZFS metadata you have to export and re-import your whole pool to get it to a writable state again.

Meanwhile fsck on a traditional filesystem will gleefully mangle data that's actually fine in face of transient corruptions.

I've had this happen more than once, both with bad RAM and bad IO controllers - previously fine static data suddenly being detached from the filesystem and appearing in little bits in lost+found, because bit flips effectively causes it to hallucinate problems to "fix".

http://open-zfs.org/wiki/Hardware#Background

This would seem to indicate that this sort of thing is exceedingly rare and similar issues would have similar effects on other filesystems.

What is scrubbing and resilvering if not a fsck equivalent?

Scrubbing corrects on disk bit flips. An in-memory bit flip (which is more rare than on disk bit flips even with non-ECC memory) can corrupt a in-memory data structure which is later written to disk to all replicas, i.e. scrub will not detect it. If this corrupted data structure is later loaded and used this may cause all kinds of problems and there is no tooling to correct it.

Resilvering (which is basically a global data verify, similar to fsck) will fix bits flipped the wrong way via error correction, assuming you've set ZFS up that way. Are you saying this doesn't apply to the metadata?

A simple scrub will repair blocks that have checksum failures, but there's is no guarantee that the checksum was calculated before the bit flipped if it occurred in a buffer being written.

Scrub?

This is wrong. But flips can do many different things:

http://open-zfs.org/w/index.php?title=Hardware&mobileaction=...

Export/import is not a solution to most of those. In the cases where it does work, umount/mount is likely all that is needed.

So is ZFS more prone to defects without ECC than let's say EXT4 or NTFS?

I would like to buy a notebook with ECC ram, but Intel doesn't care.

No, all are equally prone to errors if a bit in RAM is unexpectedly flipped. My understanding is that ZFS requires more RAM and possibly more CPU than other filesystems and those costs aren't worth it if you're going to use RAM that can't detect errors anyway.

No, no one is saying that ZFS is more prone to defects without ECC. Lack of ECC increases the risk of corruption for any filesystem. The reason you hear more about ECC in the context of ZFS is that data integrity is a key feature for many who choose to use ZFS.

To answer your question: No.

They are all just as prone to defects.

I don't understand why this is so often misunderstood.

The parent poster already stated the opposite.

ECC and ZFS are orthogonal. ECC ensures that data in your RAM is not corrupted (or rather detects corruption) it helps whether you use ZFS, EXT4, NTFS etc.

ZFS increases your data safety whether you use ECC or not, but if you have to have maximum assurance that data is fine you should use ZFS and ECC.

As far as I can tell, the ECC/ZFS paranoia traces back to "cyberjock" on the FreeNAS forums. https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...

This is correct. He considers ECC so important that he is willing to spread FUD about non-ECC behavior to try to scare people into using ECC. I think the truth is scary enough to convince people, but he does not agree.

They do. There are now Xeon E3 notebooks that have ECC. It's something new they added with Skylake for that reason. They're designed as mobile workstations, so you're gonna be looking at at least $1700 for a laptop, but good quality's always expensive.

ECC is just as important for EXT4 and NTFS as it is for ZFS.

http://blog.codinghorror.com/to-ecc-or-not-to-ecc/

At least Jeff doesn't believe much in ECC anymore.

Reply on news.ycombinator.com