Comment by lisper | Hacker News Reader

Comment by lisper | Hacker News Reader

I found a compiler bug once, about 30 years ago. It took nearly a year. This was Lisp code (the T 3.1 compiler to be exact) that ran on an embedded system to control a robot. It was a 680x0 processor running vxWorks. The robot also had an arm (i.e. a robot arm, not an ARM processor - those didn't exist yet). The symptom was an intermittent crash that happened only when the arm was moving. Forensic analysis revealed a badly corrupted heap, so the actual bug was well upstream of the crash.

The problem turned out to be a confluence of two circumstances. First there was the compiler bug, which inverted the order of two instructions when popping values off the stack. It decremented the stack pointer and then pulled a now-unprotected value off the stack into a register. Second, vxWorks did not have a dedicated stack for the operating system, so when an interrupt happened it would be processed using the same stack as the process being interrupted. When an interrupt happened right on the boundary between those two mis-ordered instructions, it would overwrite the temporarily unprotected value still sitting on the stack above the stack pointer.

It took us several months to find a way to reliably reproduce the problem, and another couple of months of single-stepping through machine code to finally figure it all out. Good times.

Ouch.

I remember I had a OS bug in my realtime class where I didn't same the ARM core's flags that get set of the compare operations.

In arm, it's two instructions to do something like `if (x > 3) { blah; }`. First do a subtract of 3 and x, and this will set the NZCV flags. If the result of `3-x` is negative, the N flag will be set.

Then do a branch if N instruction afterwards.

https://developer.arm.com/documentation/dui0801/g/Condition-...

So if there was a context switch / interrupt between the two steps, my code would take the wrong if arm. So I'd get things like

    if (x > 0) {
      y = 5 / x;
    }

throwing a divide by zero error, which was super confusing. And I could work around it by doing

    if (x > 0) {
      if (x > 0) { // For real this time
        y = 5 / x;
      }
    }

since it was very unlikely to happen twice in a row.

Eventually I fixed it, but that was fun.

I don't understand. Are you saying that interrupts clobber the processor flags on ARM cores? That seems improbable. They'd be crashing all the time.

I'm guessing (since this is what my RTOS did) that interrupts were used to preempt the running task, and the OS would need to explicitly save and restore CPSR/SPSR (the arm32 processor flags) for the interrupted process, since odds are another process would be scheduled to run next, with it's own execution state and CPSR.

I didn't issue the instructions to save and restore the flags register, which was the bug.

It's possible to get it right.

> I didn't issue the instructions to save and restore the flags register, which was the bug.

So... you were writing your own interrupt handler then, yes?

Was this CS452 at uWaterloo by any chance? If so, then I encountered the same bug last term

(CS452 vet here, Fall 2004)

Am I reading this right, you're using ARM now? We were using a Pentium PC (and that was already old for 2004, but good enough for for the course, obviously).

Yes, we're currently using ARM32, and I'm actually working on porting the course to the Raspberry Pi 4.

Reply on news.ycombinator.com