Hacker News Reader

Comment by weitzj | Hacker News Reader

And what about machines without ECC RAM? I thought this is the idea for using ZFS in the first place. Or is the ECC "requirement" only important for raidz?

View on HN

The whole ECC is required by ZFS is a bit misunderstanding.

ZFS puts guarantees that your data will be safe, but if has no power to help you if your data gets corrupted in memory. The ECC is the last piece needed to guarantee data safety.

So if you don't have ECC your data is still safer with ZFS than traditional file-systems, ECC just increases the safety further.

How does corrupted memory affect ZFS's performance? Much of the replication state is stored in memory; is it possible you could lose data from a single bit being flipped?

AFAIUI, yes. If a single bit in the root's checksum is flipped just before being written to disk you'll probably lose the whole file system.

You understand incorrectly.

http://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=2...

http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo...

http://blog.brianmoses.net/2014/03/why-i-chose-non-ecc-ram-f...

etc.

Could you be more specific? You seem to just be linking to random posts about ECC vs. non-ECC. I don't see anything specifically there about the root of the file system.

(I'll happily grant that this scenario is so unlikely as to be impossible for all practical purposes, but having skimmed the stuff you linked to I don't see why it couldn't happen theoretically.)

If I were creative, I could probably kill any filesystem with 1 bit flip if I could control the bit.

Without ECC RAM, you're far more likely to get uncorrectable / unnoticeable corruption.

This is not unique to ZFS, and it doesn't make ZFS worse than other filesystems. But since the reason you'd use ZFS is often to avoid any corruption, it's tradition to advise the use of ECC.

So is ZFS more prone to defects without ECC than let's say EXT4 or NTFS?

I would like to buy a notebook with ECC ram, but Intel doesn't care.

No, all are equally prone to errors if a bit in RAM is unexpectedly flipped. My understanding is that ZFS requires more RAM and possibly more CPU than other filesystems and those costs aren't worth it if you're going to use RAM that can't detect errors anyway.

No, no one is saying that ZFS is more prone to defects without ECC. Lack of ECC increases the risk of corruption for any filesystem. The reason you hear more about ECC in the context of ZFS is that data integrity is a key feature for many who choose to use ZFS.

To answer your question: No.

They are all just as prone to defects.

I don't understand why this is so often misunderstood.

The parent poster already stated the opposite.

ECC and ZFS are orthogonal. ECC ensures that data in your RAM is not corrupted (or rather detects corruption) it helps whether you use ZFS, EXT4, NTFS etc.

ZFS increases your data safety whether you use ECC or not, but if you have to have maximum assurance that data is fine you should use ZFS and ECC.

As far as I can tell, the ECC/ZFS paranoia traces back to "cyberjock" on the FreeNAS forums. https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...

This is correct. He considers ECC so important that he is willing to spread FUD about non-ECC behavior to try to scare people into using ECC. I think the truth is scary enough to convince people, but he does not agree.

They do. There are now Xeon E3 notebooks that have ECC. It's something new they added with Skylake for that reason. They're designed as mobile workstations, so you're gonna be looking at at least $1700 for a laptop, but good quality's always expensive.

ECC is just as important for EXT4 and NTFS as it is for ZFS.

I learned this the hard way on an old server that did not have ECC.

I had a file server happily ticking away using ext4.

Converted it to ZFS - and a week later got file system corruption reported. Ran a very extensive memory test - and sure enough I had bad RAM (but it took 2-3 days for the errors to show up).

In the wild there has to be a ton of corruption that just never gets discovered without end to end checking.

If you have a large jpgs or MKVs - a flipped bit here or there is not going to be apparent.

Why would ZFS be worse than ext4 or anything in this way if you don't have ECC?

Genuine question, I don't understand this claim. As far as I can see, ZFS provides protection against some types of failures on disk, which ext4 doesn't. ECC has no impact on that, it protects another dimension.

I read this as they've been having corruption happening all along, but only ZFS detected it?

Yes - exactly.

To be clear, ext4 did not cause the corruption - but neither did it detect it or correct it. It happily sent corrupt blocks out to disk.

I think this happens a lot more frequently than people realize.

One reason is that you generally perform scrubbing, thus you are potentially rewriting data which would otherwise be at rest. If your memory is bad, this could replace good data with bad. FS that doesn't scrub doesn't have this issue.

No it is worse with ZFS because it doesn't have a fsck tool. If you have a bit flip in the ZFS metadata you have to export and re-import your whole pool to get it to a writable state again.

Meanwhile fsck on a traditional filesystem will gleefully mangle data that's actually fine in face of transient corruptions.

I've had this happen more than once, both with bad RAM and bad IO controllers - previously fine static data suddenly being detached from the filesystem and appearing in little bits in lost+found, because bit flips effectively causes it to hallucinate problems to "fix".

http://open-zfs.org/wiki/Hardware#Background

This would seem to indicate that this sort of thing is exceedingly rare and similar issues would have similar effects on other filesystems.

What is scrubbing and resilvering if not a fsck equivalent?

Scrubbing corrects on disk bit flips. An in-memory bit flip (which is more rare than on disk bit flips even with non-ECC memory) can corrupt a in-memory data structure which is later written to disk to all replicas, i.e. scrub will not detect it. If this corrupted data structure is later loaded and used this may cause all kinds of problems and there is no tooling to correct it.

Resilvering (which is basically a global data verify, similar to fsck) will fix bits flipped the wrong way via error correction, assuming you've set ZFS up that way. Are you saying this doesn't apply to the metadata?

A simple scrub will repair blocks that have checksum failures, but there's is no guarantee that the checksum was calculated before the bit flipped if it occurred in a buffer being written.

Scrub?

This is wrong. But flips can do many different things:

http://open-zfs.org/w/index.php?title=Hardware&mobileaction=...

Export/import is not a solution to most of those. In the cases where it does work, umount/mount is likely all that is needed.

http://blog.codinghorror.com/to-ecc-or-not-to-ecc/

At least Jeff doesn't believe much in ECC anymore.

ECC is a prequisite for any filesystem to behave as intended. ZFS needs ECC just as much as other filesystem does. e.g. ext4, XFS, etc.

Reply on news.ycombinator.com