Hacker News Reader

Comment by jaxtellerSoA | Hacker News Reader

>The Intel is probably faster...

Only in specific IPC heavy workflows.

The amd probably can be built with ECC ram while the intel cannot, as well. I hope that’s a hat trick amd pushes them to release on the i7s.

When was the last time your computer crashed because of a cosmic ray?

The problem isn't a crash, the problem is silent data corruption. I have a system with 128 gigs of ECC and it catches a soft bit flip or two about every 6 months.

Most people don't have that amount of RAM. What are the chances that a soft bit flip will cause silent data corruption that matters for a user with, say, 8 GB of RAM?

The answer to the question you've asked depends heavily on the particular user and their work, of course. If the user is just playing games, who cares? But maybe they're doing financial calculations or compiling software that lots of users run or maybe they're doing aircraft structural analysis.

In any case, bit flips are much more common than were suspected: https://arstechnica.com/information-technology/2009/10/dram-...

I believe strongly that ECC should be standard, because you can't safely assume that your users are doing worthless work. Apple got this right on (non-Mini) desktops a long time ago. Not yet on laptops, unfortunately.

That study is faulty. The intern who did that study didn't know that Google would buy DRAM chips that failed manufacturers QA, but them on DIMMs themselves, and retest them at lower frequencies and with ECC turned on. When they already have to be tolerant of any node failing because of their scale, they can start playing fast and loose with this sort of thing if it makes financial sense.

EDIT: At -3 so far, does anyone want to explain the downvotes? I saw the google slides first hand, and there are comments from 2009 in that article saying the same thing.

I didn't downvote, but I'd guess that they (a) want an authoritative reference to the story you're sharing, or (b) figure the point is moot because these errors still happen, even if less often than depicted by the study, and that ought to be enough to justify using ECC.

Like I said though, just a guess.

All ram should be ecc. No one would accept this incorrectness BS if the precedent hadn’t been set by the monopolist.

It's really the memory manufacturers who are enforcing and profiting from the ECC shakedown though, AFAICS.

Intel probably gets more from selling a Xeon.

Didn't down it either, but: "The intern" is a well-respected CS professor; the paper had two Google authors whom one would expect to have knowledge of Google's oddities. https://ai.google/research/pubs/pub35162

Your comment provided no substantiation of your claim, merely hand-waving, while casting aspersions on someone else's work.

If you’ve got files you care about then care about bit rot. I want ECC so my ZFS volumes don’t silently corrupt.

> Most people don't have that amount of RAM.

Most people don't buy desktops with 16 or 32 or 64-thread processors designed to maximize throughput.

Those who do tend to want to max out how much RAM they can shove in their box.

Bit flip in the dirty cache that gets written to disk? Depends on what data structure was corrupt...

As a counter example, my desktop has had 128 gb of ram since two years ago. I've never needed that much, 64GB would have been enough for the number of vms and containers that I run. But I couldn't pass up a good deal. I got the ram for $600.

Real statement of the times that I had a tough time finding 128 gig ram sets (on a short search - this should be easy).

Run a filesystem that checksums data (e.g. ZFS) on a system without ECC RAM for a few weeks and then come back to us... I've done it, the results were surprising, and I replaced the RAM with ECC.

If you think bit flipping is rare, check out this write-up: http://dinaburg.org/bitsquatting.html

Bitsquatting: DNS Hijacking without exploitation

When bit-errors occur they can change memory content. Computer memory content has semantic meaning. Sometimes, that meaning will be a domain name. And applications utilizing that memory will use the wrong domain name.

Nice point. As with all things that follow a distribution and given the large number of machines this error will occur.

most bit flips won't cause a full-on crash, but may corrupt data in funny ways, or affect unused memory. I think people underestimate how common they are[1]

[1]https://ai.google/research/pubs/pub35162

You computer? Not sure. Computers in general? A noticeable fraction of Firefox crash reports are due to single-bit memory flips.

Huh, this is really interesting. Can you go more into this? How do you know the cause of the error? What kinda numbers are we talking here?

Also, are cosmic rays really the main source of single bit flips as apposed to just bad ram maybe?

Firefox crash reports include the contents of registers and a few instructions around the instruction pointer. If the crash is in compiled code, not JIT code, you also know where in the binary you were and can get symbols from a symbol server, then compare what the instructions should be to what they actually were.

Some of the time the instructions don't match up, indicating corruption _somewhere_.

For the specific case of crashes in JIT-generated code, the contents of registers and the instructions can be related in various ways (e.g. if you have a jmp instruction the register better contain your code location). And if you know where your code locations might be (because you're a JIT, and are generating the code and aligning it in memory yourself) and the register with the code location looks like the sort of address you would end up with but with one extra low bit set, say...

I am having trouble right now finding the bug report where some of the JIT engineers were analyzing crashes in jitcode, but about 1/3 of those were due to bitflips if I recall correctly. What that means in terms of absolute numbers (or numbers per user-hour, which would be even more useful), I don't know.

Note, by the way, single-bit flips can be a consequence of a bad memory chip, not just of cosmic rays.

Thanks! Incredibly interesting.

Reply on news.ycombinator.com