Random bit-flip invalidates certificate transparency log – again?
groups.google.com
groups.google.com
https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf
Their metric of “25,000 to 70,000 errors per billion device hours per megabit” is a bit hard to grapple with. If you assume each error is a single bit then that’s 20 to 50 bytes per GB DIMM per month, or one bit per GB every two hours.
How is that possible? Wouldn't such an error frequency lead to user-observable problems all the time? For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. Yet I've never encountered this situation in practice, nor heard of anyone else encountering it, and like most people I don't even use ECC RAM. That seems incongruent with the figure quoted above.
But I agree this sounds unrealistic.
Recently I had to troubleshoot a linux process that started segfaulting randomly. Having dealt with non-ECC bit flips before I took a copy of the binary (i.e. from the filesystem cache), dropped the caches, and compared and disassembled the binaries: a single bit flip resulted in an illegal instruction.
Cost of investigating issues >>> cost of ECC
> most bit-errors... are handled in userspace error-detection
If there really are a significant number of bad-RAM errors getting caught at the TCP layer, then there are also a significant number happening after that. There aren't, so there aren't.
Additionally, Google runs hardware in their datacenters in a much more pathological way than most consumers will (e.g. they are more likely to run hardware much hotter than typical consumers).
The point is that the figures in the paper are an interesting starting point for discussion, but they're not necessarily applicable to your laptop.
It's hard to see how this could be explained away with the slight increase in RAM densities since 2008, or different temperatures in data centers (plenty of individuals run their systems crazy hot, e.g. when the tower is stuffed with dust after years without cleaning).
> we'd be hearing anecdotes of one-character diffs appearing out of nowhere all the time
On consumer devices it might simply be a small "attack" surface. A server running 24/7 serving countless requests would probably see more issues than a laptop running idle 95% of the time and loading a website every few minutes. Also a bit flip isn't guaranteed to cause a noticeable bug - and if it does it might not stand out from the usual noise of janky software oddities.
How long would you expect to have to run memtest on any desktop computer in good condition before you observe a memory error? Because I would expect the number to be somewhere between 'weeks' and 'forever.'
That is just Windows Explorer being shite. They even have a specific “restart” button for it in Task Manager for when it leaks into itself so much it stops functioning usefully.
https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...
I have access to some massive number of API logs for my job. So it was easy for me to do something similar - most bit flip errors would result in a 4xx of some sort, especially 404.
Every time there's a huge magnetic storm, I run a query to see if 400's go up correlating with the storm. I even filter the logs for the geo hardest hit. Nope. No correlation. ¯\_(ツ)_/¯
[Mike O’Brien] wrote a module (“OsStress”) which would allocate a block of memory, perform calculations in that memory block, and then compare the results of the calculation to a table of known answers. He encoded this stress-test into the main game loop so that the computer would perform this verification step about 30-50 times per second.
On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate!
At first, I was not surprised that they saw 1% of GuildWars players had these issues. My reasoning was given the time of the article (2012) and pointing to a more distant hazy point somewhere between StarCraft and Diablo[0] their user-base probably included a lot of "custom builds", a not-small portion of which would be "built intended for overclocking", in an era when the firmware/motherboard/processors would let you apply settings that would become extremely unstable under load[1]
... and I got about that far before I thought, only 1% of users managed to make their "let's see what happens if we alter the voltages/frequencies such that we run the DDR2-333 memory at DDR2-2048 (it's been a while), change the various "latency" settings throwing caution to the wind. Meanwhile all of this hardware is sitting in a case with a fan configuration that slightly lowers the air pressure in the case[2] ...
Back in the day (less common, today) you could often put any value from a range of "Way Too Low" to "Way Too High" and steps in-between. If it could reach the boot sector,... "Weeeeeeeeee!!" I recall a case of a customer who completely trashed an Intel Pentium (2? 3?) chip via firmware settings "my. kid. was. F---in'. with![3]"
Maybe I'm wrong and it's just a matter that the game involved attracts people who rarely OC/build and the 1% is "just those people", but anecdotally "in my life", everyone I knew back then who was a PC gamer either started with a build of their own or went that route the instant they had a job which allowed them buy their own computer (rather than sharing Dad's or getting his hand-me-downs if you were really lucky/have fewer years on you than I do).
My experience, especially back in the 08-09 era, was that things were ridiculously forgiving. Depending on the use, overclocking would either "do nothing at all" or maybe let you get away with slightly better graphics settings while maintaining a playable game. Generally, I purchased the cheapest memory rated slightly better than what I intended to configure them for (DDR2-xxx slightly higher than where I will clock it) with whatever I decided were the right latency values[4], placebo-overclocked day-to-day, but would push it from time to time. There's been times the system simply doesn't boot but one "tiny notch south" and it'll pass a 72-hour memory stress-test.
I've had crashes, locally, that I can't explain beyond "memory corruption" but I can't say if it's an OS failure, hardware failure or "something that's not memory corruption but malfunctioning other hardware" ... it passes a stress test, I usually find the "other hardware" a little while later (surprisingly often the PSU, for me).
[0] Unclear if editions implied are "I" in both cases, and it's not important enough to matter since reasonably earlier than 2012 puts you in the category of "the firmware/motherboards will let you do incredibly stupid things".
[1] Sometimes subtle "some character's armor texture is transparent" (making your enemies see-through, not naked, much to my teenage chagrin). Sometimes crazy like "Everything is shades of green with partial geometries".
[2] Say, if you had to have the most sleek case where every available air-flow "vent" can take a fan facing inward or outward (where it can sometimes be a bad idea to not leave one empty) has all of the fans blowing out... I've seen "The Vacuum Computer".
[3] So, this really happened to me (CompUSA, filling in at the Parts Department), and that's exactly the way I hear it, but you need to understand that it wasn't some form of "Jersey Shore" accent -- this guy was absolutely Korean with an accent so thickand not a day younger than...60? Aside from the shock of a person very much struggling with the English language dragging out the "F" word, you just don't see a plain-looking-harmless-old-man speak with such vigor and gusto!
[4] This being "unimportant" information for me day-to-day, requires hours of "refreshing via Google" and learning every miserable thing that's changed, coming to a complete understanding, finding that the memory I want either doesn't exist, doesn't work with my mother board, or costs a fortune and settling somewhere pretty close to what is "typical" if not ... exactly typical.
I think the amount of textual user data in memory is dwarfed by various kinds of program data (pointers, metadata, static strings, libraries, etc) as well as image data. It seems pretty likely that bit flips could result in a crash once in a while or could have no effect at all if the bit flip is in some rarely used piece of an executable loaded in memory. For example, would anyone notice that a letter in an error message string is corrupted?
So errors may be happening at the rate that researchers report, but with only a tiny fraction of them happening in user data where they will be noticed.
The actual text representation is tiny compared to (for example) the framebuffer of said text rendered to the screen. I wouldn't be surprised if the size of my desktop background in RAM is larger than all the code I have written ever in my professional life. You probably wouldn't even notice a bit corruption in that, especially if it's a temporary image buffer being updated constantly.
As you'd expect a random error to hit things proportional to their size in memory, there's likely orders of magnitude more running code in your editor than the actual text you're editing, which is more likely to simply crash than silently corrupt a diff.
And on crashing, there's issues I've seen from user reports with backtraces and memory states that just don't make sense. I personally suspect that some, especially those without multiple examples, could quite easily be due to random memory corruption. And there's a long enough tail of these weird issues from that sampling I honestly wouldn't be surprised if such corruption events were a lot more common than many people think.
And that's on a pretty locked down platform so no worries about consumers overclocking. I've heard it's even worse for some of my colleagues working on more gamer-adjacent products - some people's definition of "Stable" is rather loose.
And as for the OG paper, I think it also said the errors weren't evenly distributed over all hardware - likely due to manufacturing differences some are run closer to the edge than others, but in a way that don't break quickly enough to easily test and be replaced as defective. So you might just be unlucky and have a weirdly unstable system.
I'd expect laptops and phones with their limited cooling capacity to run their hardware much hotter than datacenters with their expensive HVAC systems
But I would say that with more density you actually increase the error rate. Things like rowhammer are possible because higher density means smaller capacitors (as volume, and also as capacity) and so easier to casue bitflips. Or there is something in recent RAMs to compensate for the smaller caps?
Seems odd to me that we have rely on some paper from 2009...
Tons of people do this and have no issues with bit flips or we'd be hearing about it.
Dimms can "fail", but still have low error rates. Stories are pretty common, LinusT had one in the last year spent something like a week tracking it down, assumed it was a new kernel error. Tracking down memory errors with ECC is a painful process that involve tracking down numerous possible causes crashes.
Sure bitflips are easy to ignore, but weird things do happen. Linux crashes, processes crash, suddenly things are weird, the desktop doesn't work like you expect. People assume that "the system is buggy" or it's related to patching, or some other user space error, but sometimes it's just a bitflip.
Sadly there are few cases where a bit flip is obvious, but if you keep 10k files around for a decade it's pretty common to find a few corrupted. Various weird behaviors have been tracked down to memory issues. GCC for years had an error that nearly always was an internal compiler error ... triggered by a memory problem. One famous speed run of a game had something very weird happened, which was tracked to a single bit memory error.
Sure can you get away with ECC, certainly. But the minimal price increase is in my opinion is quite reasonable. I'd much rather get a "single bit error corrected on dimm #3" dmesg, then occasional (or not so occasional) process, kernel, or filesystem crashes.
I'm typing this from an 2015 system with a Xeon e3-1230 CPU that was cheaper than the equivalent i7, but has ECC support. Sure I spent a bit more on the motherboard and dimms, something around $120. Money well spent. I find system crashes quite disruptive, even 1 less per year is valuable to me.
Desktop ryzens support ECC for several generations, but ECC support varies by chipset. AMD forums often discuss it. Some Ryzen motherboards mention the support and certify ECC dimms.
So ECC is possible, but annoying.
I'd say that we live in a world of checksums, cryptographic checksums, retries, deterministic builds, distributed VCS (like Git), digital signatures, etc. which helps a lot with the very rare bitflips that do actually happen.
> For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately.
I fully agree. If I set say "unstable" settings in my UEFI/BIOS for the RAM, I cannot compile, say, Emacs, without getting a segfault.
> For example, in the average statically compiled codebase, flipping a single bit in source code being edited (with the code then being saved to disk) will make it fail to compile with high probability, which would be noticed immediately. That seems incongruent with the figure quoted above.
I don't understand it either.
It's not the same as "full" ECC, though, as it does not have vision once the data leaves the module (on the wire, at the CPU, etc)
Edit: nope, I got that last step wrong, it's actually 200 errors/hour/terabyte, or 1 error/gigabyte/5 hours, pretty close to what you said.
Most of the time the flipped bit will be in a cat photo and go unnoticed.
Some in unused and unsaved data, some in code paths I'm not using. However some will probably crash a program because of a wrong or invalid machine code instruction or visibly alter data.
Times that per the number of people using computers. Those spelling mistakes or wrong figures were not our fault after all?
In a code editor people should see one character git diffs in files that are open in the editor, parked in a long unused tab (if the editor detects the change and saves it or asks to.)
The dominant cause of DRAM failures is bit flips from variable retention time (VRT), where the cell fails to hold charge long enough to meet refresh timing. These are believed to be caused by stray charge trapped in gate dielectric, a bit like an accidental NAND, and they can persist for days to months. This is why the latest generation (LPDDR4x, LP/DDR5) have single bit correction built into the DRAM chip. Along with permanent single cell failures due to aging this probably fixes more than 95% of DRAM faults.
The DRAM vendors sure could do a lot better on publishing error statistics. They are probably the least transparent critical technology used in everything, but no regulation requires them to explain and they generally refuse statistics on faults even to major customers (which is why folks like AMD run large experiments at supercomputer sites to investigate, and most clouds gather their own data).
That said, DRAM chips are pretty good. The DDR4 generation probably had better than a 1000 FIT rate per 2GB chip, so in a laptop with 16GB that would have been less than 10 error per million hours, or under 1 per 50 laptops used for a year.
For many of us the vast majority of data is in media files. I personally notice broken photos and videos every now and then. I would love to have a laptop with a competent ECC level, but they do not exist. Even desktop servers often come without. It is unclear how much better the LP/DDR5 generation will be since the on-die ECC still does not fix higher order faults in word lines and other shared structures, which may sum to as much as 10% of aging faults. All simply educated guesses, since the industry will not publish.
What's the big advantage of DRAM over SRAM? In school we learned that DRAM was cheaper -- but surely the difference between 1T1C and 6T isn't more than 6 and my intuition says C is big so it's probably 2 or 3 or something for a given process generation. The problem is that the latency of DRAM is absolutely dreadful. On one hand I see a staggering amount of engineering that goes into hiding DRAM latency, and on the other hand I see that DRAM has become so cheap that many systems are over-provisioned by a factor larger than its theoretical cost advantage purely by accident. The "obvious solution" would seem to be DIMMs of SRAM with (comparatively) wicked fast timings -- but this doesn't happen, despite the fact that the memory industry is extremely competitive and filled to the gills with smart people, so presumably there's another factor that stops "DIMMs of SRAM" from being viable. Do you happen to know what it is?
> one capacitor, whose dielectric is simply the gate insulation
Every DRAM cell depiction I've seen in the last ~5 years has had a gigantic trench capacitor. Are those not in production?
MRAM and other persistent memory technologies might be used someday, but there's a lot of R&D work to get them to the same level of price and performance as DRAM. It's sad that Intel gave up prematurely (imho) on Optane.
Yeah, too bad about Optane, but CXL gives me hope that the next decade will bring more action in this space.
How many SRAM arrays exist in electronics we use every day that don't have, at minimum an error detection and reset mechanism? What about hardware registers that aren't structured as 2D arrays, how often are those protected? Things like buses and counters are at least somewhat vulnerable too.
I think that, for CT, this should be fairly straightforward. Some machine with access to the signing keys should generate new nodes and signatures and push those internally to some front-end machines. The latter (on separate physical machines) should fully validate the result before propagating it any farther. No outside user sees the result until at least, say, 3 machines fully validate it. Then, if validation fails, the state could be rolled back internally.
When a rollback occurs, the signing machine would think it’s signing a new, conflicting state, but that’s fine: no one outside the log operator has seen the old conflicting state.
In CT, all append are controlled. Nothing is anonymous.
Unlike cryptocurrency, this would require no fancy public protocols, negligible computational resources, and no additional verification overhead outside the log operator at all.
"Once is happenstance. Twice is coincidence. Three times is enemy action"
Everything is fine until these flips happen in critical code or data paths. ZFS corruption is a famous example, but there's also DNS corruption that happens quite regularly (and has been demonstrated to be usable for malicious purposes).
I'm a little surprised the certificate transparency protocol doesn't validate the incoming data well enough to detect these bit flips, but on the other hand most software I've seen just assumes the bytes in memory and the bytes received through the network are all what they're supposed to be.
I've run memtest many times, often for days, and it never detected any bitflips in RAM. And memtest is specifically designed to exercise every bit of memory and detect bitflips. While I've seen my share of PCs with memory errors, when you see one you replace memory / tweak settings, and then run memtest for a day or so to ensure it does not reappear.
Also, "unused memory" should not be a thing in modern PC: the read cache should expand to fill it - it's super fast to evict and provides tangible benefits in case of hit.
The errors usually occur on the boundaries - like in the network or in the SATA connection to disk or even in USB bus. For example the original "ZFS corruption" story (which seems t be gone from regular web, but I think its [0]) pretty clearly mentions damage "on the way to disk".
[0] https://web.archive.org/web/20091212132248/http://blogs.sun....
It doesn't have to be three machines owned by separate organizations, or even in separate buildings. Just three servers in the same rack, doing the same computations, and nobody signs anything unless one of the other two produces the exact same result.
This is getting to the point (2 log failures in just under 2 years) that I wouldn't be surprised to see some certificates invalidated because they only used 2 transparency logs and both failed within the lifetime of the cert.
Everything is terrible.
That sounds bad. What does this mean? Does the entire log need to be thrown out, and a new log needs to be create to start from scratch?
"OP here. Unless you work for a certificate authority or a web browser, this event will have zero impact on you. While this particular CT log has failed, there are many other CT logs, and certificates are required to be logged to 2-3 different logs (depending on certificate lifetime) so that if a log fails web browsers can rely on one of the other logs to ensure the certificate is publicly logged.
This is the 8th log to fail (although the first caused by a bit flip), and log failure has never caused a user-facing certificate error. The overall CT ecosystem has proven very resilient, even if a bit flip can take out an individual log.
(P.S. No one knows if it was really a cosmic ray or not. But it's almost certainly a random hardware error rather than a software bug, and cosmic ray is just the informal term people like to use for unexplained hardware bit flips.)"
What is the process for a CA to rebuild the CT log, if one exists? Is it something like what is illustrated below?
Let CA1, CA2, CA3 and CA4 be different certificate authorities. The set enumerated alongside each CA is the set of certificates logged into its CT log.
CA1 : {c1, c2, c3}
CA2 : {c2, c3'}
CA3 : {c1, c2}
CA4 : {c1, c3}
Is that kind of how it would work or would CA2 just truncate its log and restart from this point forward?
The log operator may choose to stand up a replacement log. The new log will have a different key, URL, and name from the old log. It's for all intents and purposes a completely different log.
If the random bit flips affect code -- then if the code is deterministic, then one solution (or at least amelioration) might be running multiple copies of the same code -- but loaded at different memory locations and where the results of one set of code's calculations are checked against the results of the same set of code's calculations -- but loaded and executed from a different memory location...
Kludgy? Yes -- but if the underlying hardware is buggy (random bit errors which cannot be removed for whatever reason) -- then it may be the only effective way to make the system work, despite the kludgyness...
Which brings up a strictly academic question -- what would an OS where each OS data structure and OS code path was replicated/redundant (and the results of running each redundant code path / data structure and the results checked at various intervals) -- look like?
(I know NASA did something like that a long time ago with using something like five redundant computers where each computer checks the results of the computation of the group, and if there's an inconsistency, the computer producing the inconsistent result would be shut down...)
>Over the course of about seven months, 52,317 requests were made to the bitsquat domains [0]
[0] https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...
They check where the most requests to bitsquat domains come from. They chose microsoft.com as a more neutral reference than e.g. FB. And looking at the graph, the lion's share of requests to bitsquat domains for microsoft comes from... China?? Followed by Brazil? (and only then by US)
It's incredible how a global propaganda machine has turned one of the most transformative technologies of our time into something that's considered shady and even crime-adjacent by default, and for which "legitimate" use cases are somehow considered special, when in reality there are countless potential applications for distributed ledgers. But the powers that be want to maintain centralized control at all costs, and their pushback has clearly reached even the minds of HN users.
Until then, trusting a central authority will do just fine. It works well enough now.
In a B2B setting where you have 3 parties working together on an integration project wherein 2 parties are B2B vendors and the 3rd is a B2C org, where does the centralization reside? What about vendors of the B2B vendors? I honestly don't know what we are getting at with this word anymore.
If we were to apply something like SQL Server w/ Ledger tables (i.e. centralized blockchain) to this kind of problem in my shop, we would almost certainly find a solution that all parties would find agreeable. This forces you to trust 1-2 parties (i.e. Microsoft themselves), but in the above we agree that for many (most?) areas this may likely be explicitly desirable. There are also technologies (again, centralized) that provide non-repudiation through the hosting layer itself. Example of this being something like Azure Confidential Ledger.
The part where this seems to get frustrating for people is the desired crystalline & immediate nature of the system. If you operate with the tiniest amount of extra flexibility you can get so much more done - E.g. perhaps the business can review a data tamper event tomorrow with their partners on the phone.
Blockchain does nothing to resolve this, and it seems like a much more more common issue than malicious manipulation of data on the backend. Indeed, centralized authorities are actually a kind of solution to the dirty input problem, because they can alter records to agree with reality after the fact if a mismatch is discovered.
What would a distributed agreement do that isn't achieved by the existing multiple logs per cert?
00000030: 9126 9384 ....
Instead of 9284 ....
We do a lot of extra testing on top of our vendor's to catch weak bitcells before a device is shipped to customers. Over many generations of tech, RAM faults have always been a small but constant source of faults.
But for our devices, RAM faults don't seem to be getting much better, they're always a pain point.