Hacker News Reader

Comment by jrockway | Hacker News Reader

Not having ECC is the biggest scam in computing. Ever hear of "bitrot"? That's memory errors that have been saved to disk. We have made millions of people lose their data so that servers can be artificially more expensive.

Intel was responsible for most of this. It is hard to be sad when seeing how they've lost the market lead.

View on HN

I would wager 99.99% of bitrot is silent corruption that goes unnoticed until it affects something particularly important. Without integrity and error correction in all paths along a processor, storage hierarchy, and network paths and at rest, there's no way to prove a system will ever remain reliable.

The network level is usually pretty solid in that regard with checksumming at wifi/Ethernet protocol level and at tcp, but that's the end of it for most systems.

TCP checksums are very basic. Ethernet has a proper CRC, which at least has certain mathematical guarantees about what it can detect. The TCP checksum fails if the same bit is flipped from 0 to 1 and 1 to 0 in the same bit position modulo 16.

modulo 16 really? I thought it is modulo 2^16-1 (size of multiplicative group of GF(2^16)) which is much bigger than typical packet size

The TCP checksum is based on the sum of all odd bytes and the sum of all even bytes. Adding a power of 2 to an odd byte and subtracting the same power from another odd byte will cancel out. I'm not sure what guarantees CRC-32 gives you, but they're better than that.

I wonder if it's practical to do ECC on 64 bit words instead of bytes. A 13% price increase (or capacity drop for the same DRAM price) is substantial and might justify the penny pinching, 1.5% is negligible if it leads to a similar stability increase as standard ECC DRAM. If you are often getting more then one bit flip per 64 bit word, than that RAM is garbage anyway.

ECC is done on 8-byte words. The basic form of RAM ECC is called "SECDED" ECC, and is an 8-bit hamming code that can correct one error silently and detect any two. It's not just a parity bit.

The number of ECC bits needed for any error correction level on n bits are proportional to log(n).

DDR actually operates in bursts of 4 or 8 64-bit words (+8 ECC), so you can make a better ECC using all 32 or 64 bits to cover the full burst rather than covering each 8 bytes separately.

In point of fact most L2+ caches are already ECC'd on whole cache lines, as I understand it. At the level of on-SOC integration this makes a little more sense [1] as it effectively "improves yield" by recovering marginal cache regions that wouldn't otherwise pass tests.

[1] vs. the semi-religious flame war about DRAM ECC in which I won't engage. People get nuts over this, and IMHO the actual data is awfully inconclusive.

Yes, most modern CPUs seem to do RAM ECC at the level of cache lines, too. That lets them do things like erasure coding to cover you if an entire RAM chip goes bad.

SECDED ECC is the academic example.

this a loaded question.

anyone interested in the topic should absolutely start from reading up on https://en.wikipedia.org/wiki/Error_correction_code and only then start looking into the engineering side, starting with https://en.wikipedia.org/wiki/ECC_memory, notably some benchmarks reported 25% performance hits (not sure if it's real or sales propaganda.)

I clearly am not talking about in-band ECC (storing the error detection data inside the program accessible RAM) that the 25% performance drop refers to.

> Intel was responsible for most of this.

Only in the sense of "Intel is responsible for most of computation" ... No one uses ECC pervasively anywhere, that's sort of the point of the article.

Intel has been pretty intent on making ECC server-only. I used their HEDT platforms for years and never had ECC. AMD is much nicer about the thing; if you want to use ECC on their HEDT platform, you can if you want. It's not super supported, but it's also not a $5000 upgrade. (Though my understanding is that ECC is mandatory for the current generation of Threadripper? That's great!)

ECC is fully supported by consumer AMD processors (at least Ryzen 7000, and I think earlier ones too). You need to pick a matching motherboard, most boards from ASRock will do. And you need to find unbuffered ECC RAM, this is more difficult than the previous two and is why I had to give up on the whole idea.

https://sunshowers.io/posts/am5-ryzen-7000-ecc-ram

Huh? I thought that "bitrot" was like content saved into the disk and the disk left unpowered for an extended period of time causing data loss. And I thought that the content stored on the disk has ECC on its own?

You get a corrupted bit in memory, you take and save that corrupted content to disk and no error correction on the disk level will help you.

I made the mistake of running a big PostgresSQL database on non-ECC memory once and I must say, it taught me some hard lessons.

Reply on news.ycombinator.com