> practically nobody should be running computers w/o ECC
Why? I’ve done so my whole life with zero problems.
Why? I’ve done so my whole life with zero problems.
Some years ago I built a small N3150 based home server with 16GB RAM. All was fine. But there was this one process that would always act weird after running for a couple of days. It was a Python server with long-running WebSocket connections which I had configured to use `select()` instead of `epoll()`, and for some reason it would eat up file handles until it crashed or froze, can't remember what it was, very odd stuff. I think it was running inside a Docker container and that container then behaved very strangely.
Turns out that when I then made a MemTest86 it had faulty RAM. All the data which passed through that area of the RAM and got written to disk was assumed to be corrupted.
After replacing the RAM (RMA) and doing a MemTest on it for a day to ensure that it was OK, that problem never appeared again.
But keep in mind, this was just one specific process which showed an issue after running for a couple of days, among other long-running processes.
2 >> There's also the chance that you think you had zero problems because you never got notified.
Both are likely true. I.e. Over my 40 years of dealing with computers ,it is likely there were errors which did not become problems.
I have about 20tb of unique files in my personal storage at this point (used to run photography business, plus personal videos and photos,plus movies and tv shows, plus actually important documents which are less than 3% by size).I have about 4-8 computers running around my home doing stuff (plus dozens of computing devices - phones, googles, whatnots).
I understand the overwhelming chance there were many errors over time in running computers or raid transfers or usb backups over decades. They just haven't manifested themselves as problems.
And yes, the corruption is there. DRAM is alarmingly unreliable. Because no ECC, we just do not notice it most of the time, or misattribute the problems it causes.
See https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
Also, when you download large files which you can run checksun against such as steam games, it is pretty common to find minor corruption (this i verified across many computers)
The manufacturer has RMA'd the whole set of memory sticks, but ECC would've detected this earlier and I'd have saved a couple of hours of my time.
And folks experienced zero file corruption… until they started using ZFS checksums and regular scrubbing.
I am happy with the point in the price/reliability/power usage tradeoff that non-ECC ram represents. Why force me to have fewer choices and use something that is worse for my needs in order to solve a theoretical problem that I don’t care about?
In general don't worry that for very important calculations, they are run a couple of times, at least just to verify.
Citation needed, as this is unheard of in software we trust regardless to do banking or sign legally binding documents with.
For such a minimal cost I'd rather have DIMM 3 died with a non-correctable error.
Measures are in the works there, such as the switch to a simpler and cleaner ISA.
The value of simplicity is often overlooked; RISC-V will lead to higher quality software and hardware.
But I digress, as this has no weight in whether or not lack of ECC is acceptable.
* https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o...
* https://web.archive.org/web/20091103163642/http://mail.opens...
There's no reason why any newly created file system shouldn't have checksumming, and if grandmothers don't know what's happening underneath the hood it's no worse than them not knowing how virtual memory or the TCP/IP stack work.
See also: PDF https://media.defcon.org/DEF%20CON%2021/DEF%20CON%2021%20pre...
Here's the basic set of problems:
1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume.
2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
5) The issue primarily is one of transparency. Most people don't realize it's an issue, and don't shop on it.
6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
Classical economic theory states government interventions do well for disaster resilience, for externalized costs, and for lack of transparency. Those are all places where markets fail. That means this should be regulated. We should have error correction in all storage and in all computation.
Best intervention, though, isn't a ban but a tax. Sales tax should be on a sliding scale from 1-10%, with:
- 1% for parts / support / service manuals being available for some extended period
- 1% for all software being open-source
- 1% for all hardware being open
- 1% for use of standard, interchangeable parts (e.g. batteries, power connectors, etc.)
- 1% for environmental factors (CO2, toxic materials, etc.)
- 1% for reusable infrastructure (e.g. being able to repurpose manufacturing equipment for crisis use)
- 1% for workplace factors (e.g reasonable benefits, audits/compliance, non-discrimination for target populations like ex-convicts, etc.)
... and so on
Those sorts of gentle incentives accomplish the same thing via market mechanisms without limiting freedom. For a commodity product, a 10% edge far outweighs costs. If Apple wants to make a proprietary magnetic screen connector and save 1mm of space, a 1% extra tax won't even slow them down.
This is simply not true. ECC means you need to store a minimum of 11% more bits, and at scale this means that you are paying a minimum of 11% more for memory.
Note that I am very pro ECC, and think that most computers should use ECC ram.
That +10% may become +5% after adding CPU, cooler, MB, PSU, case and it may become +3% after adding SSDs, a high-quality monitor and other usual peripherals.
While at work I have seen plenty of problems with computers without ECC, for my desktops and servers and even for some laptops, when I could, I have used only ECC memory, for the last few decades.
That has never increased the cost as much as I have saved by choosing wisely other components in the computers that I have built myself.
The main problem has never been the extra cost, but the difficulty to find them exactly when I needed them, because at retail the availability of ECC modules may be sporadic and many distributors appear and disappear, so if I upgrade systems after some years I seldom can find the same source.
I'd also expect RMA costs to be lower since you can be sure which DIMM is bad, and that it's not CPU, CPU socket, motherboard, powersupply, etc.
A ECC DRAM module contains 9 chips rather than 8. I'm not sure how you can claim that's "almost free" unless you have a very loose definition of "almost free".
>2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
>3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
>4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
ECC protects against solar flares and EMP? That's new.
>6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
2. As mentioned above, ECC DRAM modules are basically the same as non-ECC modules but with an extra chip. The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
I do. Let's do the math here. A typical computer has 8GB RAM. That's under $20 at my local retail store. We're talking a $2 price difference. That's with the methods used today. The overhead can go arbitrarily low if we adopt techniques used in nonvolatile storage and work in larger blocks of bits.
$2 is less than the cost of a tiny fraction of a crash.
> Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
A broader regulatory regime would certainly be good.
> If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
Mostly, because businesses can focus on a finite number of things. A $20 benefit which costs $2 to implement stops making sense if it means your local clinic needs to have someone on-staff who can do an ROI computation on adopting ECC. It's cheaper to buy something sub-optimal.
Again, that's where centralization and regulation helps.
> ECC protects against solar flares and EMP? That's new.
That's literally what they're there for. ECC primarily protects against bit flips, which are the result of random radiation. Mild flares will flip a few bits which ECC will correct. Severe ones will flip a lot of bits, which ECC will detect, and the computer can shut down before destroying more data.
> 1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
I'm not sure what you mean by "modules," but DRAM PCBs are easy to assemble. They can be made in the US overnight. DRAM chips are primarily made in the US (Micron), Korea (Samsung), and Taiwan (TSMC).
https://www.micron.com/manufacturing-expansion
And not all crises are military.
> The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
If all you need is memory, that's true. If you also want motherboards and processors which will take those chips, that's harder. If you don't want bit flips in your computation, that's impossible without redesigning all CPUs. Academic CPUs can do error-correcting computation at nominal cost, but there isn't a market incentive to bring that to market.
>Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
Did you know that medical reports are frequently viewed on consumer or consumer-grade hardware? Or that there is software that is classified as a class II medical device that runs on iOS devices and commodity desktops? Or that class III devices like pacemakers now often use phone apps for things like managing telemetry? It's all well and good to make sure the hardware in the OR is reliable, but that's woefully insufficient.
The power consumed by the relatively few gates that generate and verify the ECC code is dwarfed by the power consumption needed to transport bits over the PCB traces.
The extra memory cell arrays for ECC bits add some power consumption for refreshing their content and the extra PCB traces for the ECC bits add some power consumption, but this extra power is in the milliwatt range, so it is negligible in comparison with the CPU and GPU power even in a small laptop, not only in a desktop.
Most gaming desktops use overclocked memory modules that have an extra power consumption many times greater than the DDR5-4800 or DDR5-5600 ECC UDIMMs.
The ECC UDIMMs never need heatsinks, while many of the commonly used overclocked DIMMs will overheat without heatsinks.
So the claim that using ECC will increase the power consumption is completely bogus. Something like using a power-save profile instead of a high-performance profile, for clock management, or using JEDEC timings for memory, instead of high-performance timings, would diminish the power consumption by orders of magnitude more than not using ECC memory.
Right now, I power rails set to where billions of bits to never flip. If I can tolerate a one-in-a-billion bit flip, my tolerances go up a lot, and power consumption goes down by a similar factor.
Properly implemented, ECC would likely cut power while increasing reliability.
"Properly implemented" only happens in volume. That sort of optimization makes no sense for the very limited volumes ECC has today. Neither do a lot of other optimizations.
Still, those sorts of tweaks are more important for battery-powered devices. I'm not sure the impact on server power consumption would be important. RAM is not power-hungry relative to a Xeon or an H100.
Given that software can also crash, I'd go with a different approach: Redundant Array of Inexpensive Computers. For most situations, RAIC1 is good enough.