DRAM errors vastly more frequent than previously thought
blogs.zdnet.com
blogs.zdnet.com
Then throw in a high speed bus, processor and disk array....
They understand in theory but refuse to code for the possibility. Especialy when you get into HPC and there are clusters of 50-60 machines with 4Gb each, the chance of not having corrupt memory is almost 0.
Because the hardware can still detect multi-bit errors, just not transparently correct them. So you shut the machine down automatically until you get new dram installed.
Programmers _are_ coding for machines-will-fail-temporarily, but coding to "handle" random memory errors instead of buying the right ECC hardware would be insane.
Your data and your instruction set are in the same memory. Even if they are separated into different areas of memory to prevent buffer overflow exploits, it's all still in memory. Once the memory starts going, you're kind of screwed. It's the same as how -- in respect to computer security -- once someone has physical access to the machine, you're screwed.
With respect to memory errors in distributed environments, usually such environments are distributed to increased the processing power for number crunching. If you run all calculations twice and have code comparing them for acceptance, you're more than doubling your processing requirements.
But at the end of the day, it's all a matter of what level of risk is acceptable (or tolerable). There is no magic bullet to fix these issues.
Yes, the machine can and does call the technician when something fails. And iirc everything, including the CPUs, is hot-swappable. That means you can physically remove a CPU-book (containing processors and RAM) and your OS will keep running.
Quite a nerds dream, if you have the spare change...
Perhaps he was right after all.
Have you ever had fsck detect errors on a filesystem that you haven't abused? Guess what, that's memory corruption -- saved to disk forever.
I remember having a machine with especially flaky memory (memtest86 failed in about 30 seconds)... I detected it because dpkg's database was corrupted enough for it to cause errors in the application. I never even tried to save that filesystem...
In one usage, "soft" errors are ones that are 'caught' and transparently fixed by ECC, and thus have no effect (on a system that has ECC memory). "Hard" errors, by contrast, are ones that affect multiple bits and aren't corrected by ECC.
In the other usage, which I think is the more technically correct one, a "soft" error is a transient condition (bit flipped by cosmic ray, etc.) and the memory cell continues to operate normally on the next cycle. A "hard" error is where the cell is basically stuck in one state or another, and indicates that it's probably time to replace the module. I think you detect a "hard" error by looking for a series of "soft" errors, although maybe some architectures/chipsets detect the difference and report them in different ways...?
If anyone can substantiate either set of definitions, I'd be interested as well.
Hard errors are permanent (e.g. a bit is always bad) and that's when you throw the DIMM away.