Fundamentally Understanding and Solving RowHammer | Hacker News Reader

Fundamentally Understanding and Solving RowHammer

arxiv.org

52 pointsg0xA52A2A49 comments

Clarify something for me: If a DRAM chip is operated within its specifications, and suffers even a single bit-flip from this effect, is it not defective, either by manufacture or by design? Memory has a profoundly simple job, to return the data that was stored. Returning something other than what was stored is pretty fundamentally wrong -- either it needed more refreshes, or the specification should've prohibited repeated access patterns, or the memory controller allowed access patterns that were prohibited by the specification.

Are we all pretending this is mysterious and novel just because it would be uncomfortable to admit so much has been so wrong for so long?

If the engine in your car stutters once in a while… is it defective?

If your LCD monitor has a dead pixel, is it defective?

That is a question of quality.

Rowhammer showed that what previously was acceptable quality is no longer so.

If the engine in my car runs fine on whatever gas for 10 years, but when I put the same gas in the tank 50 tank-fuls in a row and suddenly the engine turns into a pink flamingo, yes it is defective.

If my LCD monitor displays any image I want, but when I try to show an image of a checkerboard it loses sync with the incoming signal, yes it is defective. (There were a number of similar problems in the telephone network around the introduction of T1 signaling, and it's why the 1000Hz milliwatt test tone was changed to 1004Hz.)

These things were never acceptable.

The difference in this case is there’s a malicious party acting against the system too. It would be like one gas station you go to throwing some sugar in the tank just to mess with you.

> If your LCD monitor has a dead pixel, is it defective?

Yes.

Dead pixels happen, it doesn’t mean the whole panel is defective.

It's the stipulative definition of "defective" as a term. One could assess it with functional and nonfunctional criteria. Consumer A demands perfection for full brightness video in darkness while customer B requires serviceability only to see a large letter X or O.

So you always buy Class I monitors, guaranteed zero dead pixels, not the affordable Class II monitors with up to 15 dead pixels? [1]

[1] https://en.wikipedia.org/wiki/ISO_13406-2#Pixel_Fault_Classe...

I must admit, I don't pay attention to that. But my monitors don't have dead pixels, and if they had dead pixels, I would absolutely consider that a defective monitor. The class presumably just determines whether I can get the manufacturer to fix it for free or not.

Yes, but they're also choosing the monitors that pass QA best. It may be that all of their monitors have zero defects, but that is statistically unlikely.

Look at chip manufacturing. They really care about high yields, but there are always defects, 20 years ago and today as well.

It's an issue with manufacturing in general. Perfection is typically impossible to achieve. Maybe not always, but typically.

Back to the main issue, since modern monitors are quite high resolution, you and I might both have a few dead pixels, or more likely a few that have pixels that are half-dead (dim) in one channel (of red/blue/green), and we just don't notice on our desktops -- and it would usually take careful examination to spot them even on a pure white screen.

Your mileage may vary, of course.

Dead pixels which are just dimmer than usual is okay, some variance is expected. But if a monitor has that kind of dead pixel that's stuck displaying a bright white or green or something even when the rest of the screen is black, the screen is 100% defective, whatever the manufacturer's QA standards might say.

As an EE, I’m shocked computers work at all, and so reliably. If you knew how the sausage was made you would be horrified.

From Dell:

A display with 1 to 5 fixed subpixel is considered normal and within industry standards

https://www.dell.com/support/kbdoc/es-es/000126004/dell-disp...

Yeah, so Dell sells defective monitors and doesn't serve them under warranty. Doesn't mean they're not defective.

The cost of increased performance and efficiency is unfortunately met with an increase in overall fragility of the system. As gate sizes shrink down they’re more susceptible to fluctuations in voltage, temperature and electromagnetic induction.

So yes, you could have a very robust system, but you’d have to pay more for less RAM, which market forces have chosen against.

Fundamentally irrational market forces, we might add. No person compares spec sheets and chooses the chip with a 10% efficiency advantage but with a rowhammer style vulnerability instead of the 10% less efficient chip without a rowhammer style vulnerability. At least I've never seen "is susceptible to rowhammer" in a spec sheet. So as consumers, we don't really have the choice to select the more expensive or less efficient but more robust chip. Hell, ECC isn't even an option for most classes of CPU.

Why not? There are CPUs being designed that do floating point math “wrong” but faster…

I’m sure many people would use RAM that’s cheaper and faster but row hammer vulnerable. Just as many people drive cars with engines that will break if driven at redline for long periods of time.

It’s not like row hammer is a realistic use case. No algorithm depends on row hammer access patterns. Just like people don’t buy cars to bounce off the rev limiter.

"Why not" what? Why we don't get the option to buy rowhammer-resistent RAM? I don't know, ask manufacturers and resellers I suppose? I don't make the spec sheets.

If you run a server farm, you absolutely know these things. And you are using ECC based RAM.

Most of us don't run server farms.

Most of us don’t care about rowhammer then either.

You have the option, you just choose not to pay for ECC ram.

That is fundamentally different. Floating point computation not nearly as always needs to be done to the last digit of precision. For a lot of application 10^-8 is more than enough, for some even less digits might be enough. But fast calculation might be a lot more important if you're in a real time scenario or if you're doing HPC.

On the other hand, is not only row hammer attacks that cause bit flips, random bit flips happens, cosmic ray or decline of radioactive particles might cause one. And there is _no way_ to know if one happened on non ECC RAM (I know, ECC only reduces the probability of not noticing it, but it reduces that probability), and given that there is not way to respond.

If one random bit flip happens, best case nothing happens. If you are unlucky, maybe your system will crash. If you are really unlucky your data might end up corrupted and the OS might not even be able to detect it. Or maybe you were running a 100 hours calculation, and it might go wrong in the first minutes because of a bit flip and your simulation is ruined.

Floating point operations is about precision, witch can be studied and decided based on the needs of the application, bit flips are about data corruption.

I'm guessing more crashes occur because of bugs in software or RAM / CPUs that are already damaged than row hammering.

No they are fundamentally rational. If some memory ended up making computers crash a lot and corrupt data because it regularly was flipping bits this would eventually create a bad reputation and it would be avoided over time. Rowhammer uses fundamentally unnatural access patterns to flip bits which never occur outside of running the rowhammer test so the market places a rounds-to-zero value on mitigation. And despite that DDR5 is still much less vulnerable (although it’s unclear if that will change) to rowhammer despite higher clocks and densities.

How can it be fundamentally rational when you're making choices without critical pieces of information?

And why are you discussing crashes? Rowhammer is usually considered a security vulnerability, not a general reliability issue, right?

I'm confused by what you're trying to say.

ECC deniers are quite difficult to understand. Once one has experienced an unreliable system with a few bad DRAM bits that caused data corruption, the 20% premium for ECC is well worth it. An hour of my time being wasted on debugging a wonky system more than pays for that.

Because economic rationality doesn’t require perfect information. And I’ve yet to see any practical rowhammer implementation in the wild which could target the browser.

"Economic rationality doesn't require perfect information" may or may not be true in the general case. But if I want a system that's not vulnerable to rowhammer, and I'm buying new RAM sticks, the fact that I can't know whether a candidate RAM stick is vulnerable to rowhammer or not means I can't make rational choices, no?

memtest86 absolutely can find rowhammer failures, but beyond that even if you specifically don’t know how to do that, others do and publish their findings and if some ram was really worse than others in a meaningful way here that information will be available as a noisy signal for ram purchase choices.

Most people don't do that much critical stuff on their local computer, as all their critical data is stored by Google or Microsoft and they just transiently interact with it via web applications. What they do do on local computers that could be at all complicated is play games.

The goal of a computer for the majority of users (aka, not us developers) is thereby to be as cheap as possible while being just stable enough to not fail so often as to be too annoying, while preferably being as fast as possible under highly constrained conditions to play games at high quality.

It is unfortunate that chip making is so complex and capital intensive that a disruptor company would struggle to move in and take advantage of this opportunity. To do so competitively would require deep pockets and deep expertise, both not well-aligned with disruption. But you’ve pretty clearly outlined a product feature that is not discoverable in the market today, which means that a disruptor who could get off the ground ought be able to differentiate by making this discoverable. Doing so would quickly lead to price discovery and if the margin were there, the rest of the dram producers should fall in line fairly quickly to provide this information for their products as it suddenly becomes an axis they can (must) compete on. At least in a perfectly spherical frictionless market anyway..

Edit: downvotes? Really?

What about error detection and correction? Can it be used with the more fragile kinds of Ram?

Yes, but it costs more (in $$ and performance)

Is there still a whacking great price premium between ECC and non-ECC memory for desktop, but not server, motherboards? If so then that suggests that there's more going on here than a simple expression of consumer choice.

>Is there still a whacking great price premium between ECC and non-ECC memory for desktop, but not server, motherboards?

Yes for both.

Price: On pcpartpicker a stick of 16GB DDR4-3200 memory without ECC costs $28.99[2], but the ECC version costs $52.99[1]. Even factoring in that ECC ram comes with an extra chip, it's still 62% more.

Performance: using the same links above, we see that ECC ram tops out at 3200 MT/s, whereas for non-ECC you can get 4000+ MT/s. The brand selection is also better, which is critical if you want to get high performance memory chips like samsung b-die or micron E die.

[1] https://pcpartpicker.com/products/memory/#b=ddr4&E=1&sort=pr...

[2] https://pcpartpicker.com/products/memory/#b=ddr4&E=0&sort=pr...

> Performance: using the same links above, we see that ECC ram tops out at 3200 MT/s, whereas for non-ECC you can get 4000+ MT/s. The brand selection is also better, which is critical if you want to get high performance memory chips like samsung b-die or micron E die.

Notable is that the max speed of 3200 MT/s for ECC is due to that being the fastest JEDEC standard for DDR4, you're not getting the XMP-profiles to run them faster and will have to resort to manual overclocking.

I was thinking about doing error correction in software at driver/kernel level as opposed to using ECC RAM.

Even though ECC contains the logic of doing error correction in one chip, that doesn't justify double prices.

> about doing error correction in software at driver/kernel level as opposed

I have no clue how that works without significant performance cost

As a sibling said, rhis would certainly come at a huge performance cost. Inline/Inband ECC is showing up again recently, which might be of interest. It comes at a performance cost too, but basically you use regular RAM, and the memory controller issues extra read/writes for the ECC data separately from the rest of the data for a given address.

If that gains traction, it could bring ECC to the masses.

Yes. If I could find the original rowhammer thread here, I believe I said so at the time.

For me, rowhammer has always been the most badass exploit I've ever heard of. I mean, flipping bits in memory by forcing an electrical current down a pathway over and over again? That's completely mind boggling. If I was hacked using rowhammer, I'd be so impressed I'd give the hacker whatever data they want and maybe even throw in a credit card

It's not magic. Slower but cheaper than SRAM cells, DRAM-derived logic needs to refresh periodically and partially/fully when read. It's likely an inability to model worst-case current needs or clocking accurately enough to match reality when a large number of cells reinforce a draining property.

Even ECC modules are vulnerable.

Takes longer though.

IIR - So long as the time it takes is longer than the periodic DRAM refresh interval, the ECC can effectively stop rowhammer.

https://ieeexplore.ieee.org/document/8835222

> . In addition, we show that, despite the non-trivial constraints imposed by ECC, ECCploit can still be powerful in practice and mimic the behavior of prior Rowhammer exploits

Just like with non-ecc dram, quality levels vary per dimm, and you can avoid this with good quality ecc dram.

I didn't read the full paper, which is very interesting, but on systems with proper ECC configuration[1], the scanning portion of the process would generate a huge number of machine check exceptions. Having run a large number of ECC enabled machines, high volume ECC correctable errors are disruptive to normal processing, and the machine goes out of service for repair quickly. If I were running a multi-tenant cloud, knowing that RowHammer is a thing, I would migrate the VMs on a machine throwing correctable errors, and also track the customers. VM migration during scanning is going to result in inconsistent data, and successful exploitation seems unlikely. Tracking the customers, allows one to prevent repeat rowhammerers from accessing your systems.

That said, it's still very interesting and many systems have improper ECC configuration. ECC without reporting isn't going to mitigate RowHammer; nor will it help to have ECC with reporting, if the reporting is ignored, etc.

[1] I didn't make it to their system configurations, but I did see their description that only one system under test properly responded to errors, by logging errors, and crashing on uncorrectable errors.

Reply on news.ycombinator.com