I'd say it depends on the use case. For business storage servers, especially "mission-critical", sure. (Then again, this data was created somewhere; presumably mainly on laptops without ECC. Why is that not a bad idea?)
But for a home storage server, ECC is no more critical than buying a UPS, shielded SATA cables, enterprise-grade drives, a high quality power supply, etc. etc. Together these upgrades will easily double the cost of the storage server. Since you'll want an off-site backup solution anyway in case of disaster, why bother paying for an expensive server that only reduces the chance of needing to restore from backup from 0.001% to 0.00001%?
If everyone used ECC RAM, it would be marginally more expensive than regular RAM, but it would let us (theoretically) cut the number of real-life crashes by a huge degree. I think if every computer system were like this, it would have non-obvious positive effects on society (like the ability to safely give computers increased responsibility).
Do you have a source for that claim? I know people have done tests with running large molecular simulations on hundreds of GPUs with/without ECC for days on end, and who've come to the conclusions that ECC probably isn't worth it [1].
And this makes sense. Most of a program's memory usage is data. Thus the rare event of a bitflip is most likely to occur in data. Such a bitflip doesn't cause a crash in a consumer system; maybe it changes the color of one pixel in one frame of a video file, or adds some other tiny noise somewhere.
The question is then: would reducing one crash per lifetime of the machine on every ten machines be a huge improvement?
I think you understand this but in case it's not clear to others (I wasn't sure until I got to the end of the paper) all of the machines in their study are using ECC memory. This allows them to measure both the number of Correctable Errors (single bit) and Uncorrectable Errors (more than single).
I think you are right that the chances of a single bit flip for a randomly chosen machine are about 25%. In Table 1, they say CE incidence for a randomly chosen machine is 32% per year. But they also show that a machine that has one correctable memory error is much more likely to exhibit many more.
I think that "CE Median Affct" (also in Table 1) says that among the machines that have been affected by at least one correctable error, the mean number of errors is 277. Thus rather than one crash per 10 machines over 3 years, it's likely that you'll get hundreds of crashes, but with one (or zero, or two) of those 10 machines responsible for all of them.
The Uncorrectable Error rate they found was about 1/20 of the correctable rate, a little over 1% per year. Assuming the same "use time" factor applies, this would mean that with ECC none of the 10 machines would be likely to ever crash due to bit flips, reducing the number of crashes by hundreds.
But this doesn't tell the full story either. Maybe it's best to view it as 1000 employees, each of whom is issued a laptop. If you buy them machines without ECC, the initial cost is less but 100 of them will experience frequent crashes during the lifetime of the laptop, leading to frustration, possible lost work, and early machine replacement.
If you pay more up front for ECC, the number of affected employees will be reduced from 100 to 5. How much extra one should be willing to pay for this reduction depends on circumstances, but I think the impact is quite different than merely avoiding 100 equally distributed random crashes over the course of 3 years.
I mean, how many companies do you know that give all their workers Xeon-powered laptops? My impression is that 99% of people have i3/5/7 powered machines at work.
Are really companies experiencing frequent failures caused by RAM on 10% of their employee's machines? Can anyone with such knowledge weigh in?
While reducing bitflip on all devices would be a nice step forward for consumer electronics, pragmatically I have to agree with you that it would have negligible impact in the real world. Those that need it generally already pay for it, and those that don't already find countless other and more frequent ways of breaking their machines.
[1] https://en.wikipedia.org/wiki/User_error#Acronyms_and_other_...
I've seen users place PC and Macs next to radiators or surround their device with folders and reference books, oblivious to the fact that computers can overheat and need ventilation.
I've seen users trying to use Excel as a database, then get impatience when Excel starts hanging or Windows starts running slow, and retaliate by pressing more buttons more rapidly thus locking the entire OS up.
I've seen users who would just switch the power off at the wall even night. Overtime NTFS would corrupt to a point where random crashes would happen.
I've also seen people physical attack - kick and thump - their computer when it runs slowly, an application crashes, or just through bad temper. This can damage the HDD and lead to more crashes.
There are also users who think they understand computers, so they like to tinker with lower level settings (IRQs, paging, CMOS settings, driver options, etc) then get surprised when things behave unexpectedly.
Most recently though, I had to explain to one guy who installed his own motherboard and accidentally cut a deep scratch into it, why his random crashes are potentially his own fault.
So I definitely blame users for most PC crashes. Even with the few number of kernel panics I've had over the last 15 or so years, I can attribute most of them to myself. eg I was experimenting with undocumented Windows APIs to change low level behaviors. Or I was experimenting with running non-supported file systems as my main OS drive.
What's the worst that's going to happen? Some photo is going to get a single bit error? So what? The data doesn't need super high fidelity. What about a file system crash? Well that's what backups are for.
[] I would also like to take time that I found the FreeNAS community completely horrible. They're focused solely on people setting up like HIPPA compliant scientific data, or something. It's ZFS for everything. Backups to USB drives are stupid, and you're stupid for having that as a backup strategy, and instead should drop another $1000 on a second NAS for your house, and then backup across a network.
OpenMediaVault's community on the other hand is much more helpful and open to both professional and personal use cases.
E.g. with rsync, you use the --backup and --suffix flags.
A fact that complicates this argument is the prevalence of compressed data formats. A wrong bit in a photo could corrupt the metadata, or a block of DCT coefficients, such that a whole image block or scan line does not decode, effectively ruining the photo. A wrong bit in a .tar.gz could well corrupt the entire archive. You never know where that bad bit will land.
(Yes, this has happened to me -- a single bit flip, probably from RAM error, corrupted a large .tar.gz -- I tracked it down to the bit that changed because I had another copy.)
I don't follow the reasoning here? It's not like ZFS gets worse if you're not using ECC RAM. Rather it's: ZFS is reliable enough that using ECC RAM becomes worthwhile. ZFS is still more reliable than ext4 even if you're not using ECC RAM.
In fairness to the FreeNAS community, USB drives often aren't a good backup solution:
-> USB thumb drives break all the time. They're terrible choice for any long term storage plan. So you're better off with a HDD / SSD
-> Keeping a USB device inserted all the time means the backup is permanently accessible. This is great for convenience but terrible for a backup as malware (eg ransomware) could access and write to your backup. And even if your backup isn't mounted, it could still be subject to any other physical disaster that knocks out your main archive of data (eg flooding). Backups should always be kept separate - ideally offsite.
-> So now you've got a removable HDD that you need to connect and disconnect frequently. You better buy something reasonably rugged because cheaper drives might fail due to frequent manhandling. Which means you're no longer looking at a budget device.
-> Finally, USB backup solutions cannot be automated. So there's always the risk that someone would let their backups grow out of date. If not out of forgetfulness then just out of plain laziness (eg "I'm too busy to do a backup right now, I'll do it 'tomorrow'").
So while USB storage can be workable as a backup solution, there are a lot of caveats that would invalidate the whole point of a backup if one isn't careful.
I mean if you have an 8TB NAS, and you want to use an 8TB USB drive to backup said NAS locally, I'm not sure I see that as a problem. Backup doesn't HAVE to be significantly more durable than the source material, which is why multiple backups and at least one off site are recommended.
After my time doing support for iomega (so long ago), I don't consider anything really backed up unless there are at least 3 copies, and at least 1 of them is offsite.
The mid to high end stuff are. The lower end is less so. But you pay a premium for decent storage capacity on a decent thumb drive, so either way, you're back to an external HDD / SSD drive.
> "and other USB storage effectively an HDD in an enclosure with a USB interface?"
Indeed. Hense why I said "So you're better off with a HDD / SSD" when referring to other USB storage devices.
> "I mean if you have an 8TB NAS, and you want to use an 8TB USB drive to backup said NAS locally, I'm not sure I see that as a problem. Backup doesn't HAVE to be significantly more durable than the source material, which is why multiple backups and at least one off site are recommended."
I don't have an issue with USB storage per se, I was just saying there are caveats to consider. And while you're right that backups don't have to be more durable than the source, you have to bare in mind that the kind of people who would be looking for advice about using USB devices as a backup solution would likely be the same kind of people who wouldn't have multiple USB devices nor the kind of people who would test their backups to ensure the medium hasn't degraded. They also wouldn't likely be the same people to keep their backups offsite. As these kinds of checks and additions can add significant costs.
Or to put it another way, if one is unwilling to spend $1000 on a second NAS then they are unlikely to want to spend $1000 on a few decent external drives. So that person is going to start scaling back their requirements (cheaper drives, fewer drives, etc) and quickly end up in a situation where their backup solution is total garbage.
Bare in mind, the kind of people who would wander onto FreeNAS's forums looking for backup advice are unlikely to be people like you and I who understand how to implement these things correctly. So it doesn't surprise me that many members of the FreeNAS community veto recommending USB drives knowing how easy it would be for someone inexperienced to get things wrong (eg leaving their USB device connected forgetting that some malware would just write to USB device as well)
But if the problem is the RAM or CPU, the bit could also flip before checksumming. Then the incorrectly written data will have a matching checksum and the corruption can't be detected.
That's why ECC is important.
Flipping it around: what's the point of backing up to a remote server if the data you're backing up is itself corrupted?
I don't think most of the upgrades you mentioned are really in the same bucket as ECC DRAM because only the ECC DRAM seems likely to prevent data corruption. Failing drives and cables and power loss, even failures that corrupt data, should be survivable events assuming drive redundancy, particularly given a filesystem like ZFS that will detect and repair the corruption.
But notice that a bit-error in a text file in plan storage has some consequences. That same bit-error in a text file in a RAID (or encrypted storage) has very different consequences.
Some bad ideas are just worse than others.
Having an error on one disk, versus multiple disks in a RAID totally matters.
Serious question: Do they not teach Hamming codes in undergrad anymore?
Writing corrupt data once or writing it a thousand times redundantly with parity data looks exactly the same, if it's allowed to happen. It's a valid write of invalid data. Either way, the data you have on your cheap laptop drive or your six-figure storage array is corrupt. This is why the ancestor commenter is correct about the storage technology being irrelevant to the choice to use or not use ECC.
I haven't been in school for many years. They taught Hamming codes, but they also taught garbage-in, garbage-out.