- An OS would write the "please wake me up on the next interrupt" to a hardware register, then a shadow RAM location. Interrupt happened between the two writes, so the task scheduler wrote the wrong value to the register and stopped. Finding the problem: Two weeks. Fixing the problem: Swapped two instructions and rebuilt. Write-only registers are evil badness.
- USB controller with a temperature problem: It would also work first thing in the morning, and after lunch. After maybe four days of correlating this the "Aha!" was visible on my face at lunch and I left the table rather abruptly. The initial fix was a bag of ice on the chip. Real fix took several weeks of increasingly loud conversations with the chip vendor.
- Cache on a disk drive had a bad bit, occasionally corrupting file contents or (even more fun) directory structures. Memo: All the ECC in the world won't help you if your I/O system is flappin' in the breeze. End-to-end checksum anything you truly care about (in fact, we caught this one with a Merkle tree).
Never mind the heap corruptions by code you don't even have source for, the async callbacks that fire long after you've nuked the objects and assumed everything was dead, the DMAs that came from outer space because you set them up hours ago and they finally fired, wiping out something random every time, the incoherent cache coherency systems, the serial ports that drop bytes if you send too fast . . . it's amazing anything sufficiently complex actually works.