Hacker News Reader

Comment by userbinator | Hacker News Reader

Having worked with embedded systems before, I've seen my fair share of bugs like this and it's almost certainly one related to timing/race conditions with external events. Looks like this one is no exception. An instruction trace (or a logic analyser logging the data/address buses) is extremely useful, since you can mentally execute the code that it's tracing and see where it went wrong. Configure it to start/stop logging at the appropriate points and you won't have millions of instructions to look through. Binary search is very helpful.

In my experience the truly insanely irritating Heisenbugs are those that involve physical phenomena like marginal logic levels/crosstalk/noise, and in that case even attaching (or in one case removing) the extra debugging hardware can cause it to disappear.

View on HN

Once, while writing a new program-loader-and-debugger/"bios" for a Z80 clone, I had a particularly nasty heisenbug.

The problem was my new code "usually" worked. About 25% of the time it would simply fail to do anything (it worked perfectly the other 75% - "umm... aren't computers supposed to be deterministic?") Of course, every single time I did anything to try extract any useful information from the device, it would work 100% of the time. Even tossing a quick 2-byte sequence to flip an I/O pin made the problem completely go away.

The problem was, of course, that the hardware wasn't quote done rebooting, and needed to be left alone a tiny bit longer after hitting the RST pin. About 8 cycles longer than when the hardware engineers assured me it would be "completely safe". Any time I added debug code, it simply worked as the needed delay, and was solved by a very carefully documented sequence of NOOPs.

What made this particularly infuriating, though, was it's seemingly arbitrary sensitivity to when the real code was started. Depending on how much you padded with NOOPs, several behaviors were observed.

    #NOOP | Behavior
    ---------------
      0   | original 75/25 ok/fail split
      1   | always failed
      2   | always worked
      3   | always worked
      4   | worked ~95% of the time
      5   | ~80/20 ok/fail split 
      6   | ~50/50 ok/fail split (not a lot of data)
      7   | *almost* always worked (I think I saw it fail once)
      8+  | always worked

Of course, the various stuff I would add for debugging tended to line up with the cases when it DID work.

sigh

I miss working with that chip. Z80 assembly is kind of fun. Well, except for when my boss told me my new loader had to work on our slowest 4Mhz part. Receiving a program over a 115kbaud serial line, verifying CRC-8, and writing it to flash is pretty hard when you have to do it less than ~100 clocks/byte....

On some x86 PC BIOS, there is a do-nothing delay loop (I've seen 20K and 64K cycles) not far from the reset vector, and I had figured that it was there because of this exact same reason; to give the system some more time to reach a stable state. If the voltages haven't completely settled, or there are still some undefined levels somewhere, then the CPU might be able to execute some if not most instructions correctly, but behave incorrectly for others --- overclockers are probably quite familiar with this "analogue" failure characteristic.

Reminds me of the Read-Modify-Write procedure on a lot of PIC devices. Our teacher taught us to randomly place NOP instructions until it worked; we all thought he was crazy until we realized what he was working around.

(The actual solution was to write to the LATx registers instead of PORTx)

This one is just code that the author didn't understand. He probably wrote it, but didn't understand it. Actually

call _ jr z,_ xor a ret _

takes 4 bytes on a z80. Should have done something like

call _ ret z

which saves three bytes (ret z is a one byte instruction). I guess he may have forgotten about the conditional return instructions.

And, key debounce is important. In future, when the key switches become (when, not if) marginal, there will be a lot of make/break contact events. Software debounce was very common for this class of system. Missing that, and the effect this had on the kernel being written? Very sloppy work.

3 weeks to track this down? Back in the 70's when the z80 was new we typically HAD NO DEBUGGER (unless you wrote it yourself), And, the hardware being developed may have been flaky. Hell, even the KIM-1 needed software debounce.

Author here. The `xor a` was important - it pretended no key was pressed. I'm familiar with conditional RET.

Thanks for the rest of your kind words, though.

I've experienced plenty of similar bugs that don't manifest when running under a debugger while working on modern ARM systems (not that any of it is necessarily specific to ARM, it's just what I work on).

Usually it's one of the following causes:

The boring one which should show up as a warning in the compiler: use of code with undefined behavior, which could be something as simple as an uninitialized variable or in manually written assembly use of an instruction in a context in which it has undefined behavior.

Bugs caused by poor understanding of the memory model: ARM has a weakly ordered memory model, which allows for various hardware optimisations, but the (systems) programmer has to understand it. The basic idea is that memory accesses could be observed in different order in different coherent cores, so synchronisation points require carefully placed barriers.

Bugs caused by not accounting for the modified Harvard architecture: the data and instruction caches aren't coherent. If you do JIT you'll notice odd behavior soon enough when some executable code is overwritten, but if it's just one time dynamic code generation, there's a fair chance most of the time there won't be any stale data in the instruction cache. Until there is, you get odd erros and if you inspect the code in the debugger it will look just fine, since it's fetched through the data cache.

DMA bugs: if the DMA peripheral isn't connected through a cache coherent bus, it can be quite tricky to ensure you won't accidentally transfer stale data.

Finally, something that's not so much influenced by using a debugger, it's just that the debugger is completely useless: silicon bugs. Just a few weeks ago, I found a bug in a MCU which causes it to send multiple copies of an SPI transfer for certain ratios between the peripheral's clock and the core's clock. It was relatively easy to find since SPI is an external bus which I could probe with a scope, but if a similar issuue would happen on chip, it would be a lot more tricky.

One of the most irritating ones I found was in an FPGA. There was a bug in the FPGA code with an initial state which would only fail to initiate an ADC converter correctly if the temperature was somewhere around 75F.

So at room (lab) temperature, it was fine. Out in the "field", which was 1000 feet away from the lab, nothing would work.

Did you run a post-PAR (timing) simulation on that? On designs with low timing slack or large area occupation this is really important, otherwise you may easily violate setup/hold time constraints in what are usually very nondeterministic ways. This is, in fact, one of the reasons why ASIC development is ~80% verification.

We had a bug of the kind once, where a Microblaze core would fail to boot the Linux image after what seemed a trivial modification in the peripheral code, but it turns out we were packing the logic so tightly that this change blew the timing on some critical net. Ah, the joys of digital design.

I just want to point out that nowadays people don't usually catch timing issues in digital designs with simulation. It's mostly done by static timing analysis tools such as PrimeTime, which is much more accurate, efficient, and thorough than running logic simulations with annotated delays.

Reply on news.ycombinator.com