Hacker News Reader

I forgot to mention the best part of it. The fix was a one character code change (to enable a feature that we have disabled before).

Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

>> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speeds slowed. The first circuit to fail was something in the branch condition logic. This caused all conditionals to not branch, which lead to running all code in the memory space in order. That included a few instructions to.... fire the airbag. It was a rare thing, but a very big deal.

I wonder if anyone else instantly gets a visual after reading your story:

“Business woman on plane: Are there a lot of these kinds of accidents?

Narrator: You wouldn't believe.

Business woman on plane: Which car company do you work for?

Narrator: A major one.”

I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.

And how brittle the borders we programmers see a strict, as a branching condition, "this should never happen". But in hardware, things are different.

How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?

Maybe put some code directly before bag deployment code that disarms the system. So even if the execution reaches it directly through the branches, nothing will happen

Sure but what if the address bus glitches past that block?

Maybe put arming code somewhere completely different?

Of course it's possible to create more and more improbable scenarios. That's not to say those never happen; I've seen really improbable occurrences.

All the stuff that counted on our board had brownout protection. And if it didn't it is relatively easy to deal with (as long as you are aware of the problem).

(We may have learned a bit about noise and improved further iterations of the device, but sw change was enough.)

I think something similar happened to my sd card at a hackathon years ago.

We couldn’t change the data on it. Linux would report that data was written to it, but when we unmounted and remounted it, the data was unchanged.

https://devpost.com/software/xcaliber

Oh god, that's insane

epic debugging story

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

>The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died

That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.

Once you have 700 units in the field, one failure per 700 days is no longer quite so rare.

And that’s running the particular op repeatedly, so in practice it’s an even rarer event. However with “millions of them in the wild” it doesn’t matter. Scaling is tough.

We had to clean out a small warehouse and repurpose it for testing rig. We have created a piece of hardware to do automated testing and recording, and then we created a small production chain to and produced 100 of them.

Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).

from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.

Yes, it was separate chip (per description). Yes, it was SPI flash. And yes, the word was 16 bits.

1. What does choice of MCU have to do with it?

2. Noise can definitely affect other components. Traces are antennas that both broadcast as well as receive signals.

As technology marches forward, the voltages and tolerances for communication become smaller and tighter. At the same time, you integrate more functionality on the board so maybe your sensitive MCU and flash and ram are all living close by to some radio equipment and PSU and whatever else. Power interruptions to certain components can create momentary flashes of noise that can act as EMP bomb going off. On other occasions a noisy, badly designed high frequency power component can emit noise that is picked up by some random traces due to their geometric configuration and length.

Your comment suggests you never worked on a large project with thousands of components and dozens of major chips?