Hacker News Reader

Comment by mc3 | Hacker News Reader

I hate questions like "What's the hardest bug you've ever debugged?". It means as I work I need to keep a scrapbook of "will what I did today make a great interview answer in 5 years time". I'd rather just fix the bug and forget about it.

View on HN

I kindof agree. My memory doesn't work like this, I could spend a week troubleshooting, following various twists and turns and eventually finding a solution with a big aha! moment, and not remember much of anything about it a week later, at least in terms of spontaneous voluntary recollection.

And you don't remember any bugs that took much longer to get to the bottom of than the others? Or bugs that required debugging at a much lower level than usual? Or bugs that involved a huge black box that required a lot of guessing to fix?

So my first thought was the same thing, and so I started thinking "What would I say"? And while I started with "Hmm, I don't really remember hard bugs; I remember times I spent days, and the lessons I took away, such as the importance sometimes of putting a problem down and sleeping on it...I guess this one time when I had to figure out why something was so slow and eventually tracked it down to an O(n^2) implementation when it could have been linear was cool...oh, but wait, we actually had something here where due to mutable state we were growing a list without end and it was being sent out across the network, and we hadn't tested it in staging because it was just for metrics, and that ended up taking out a downstream system in prod" and...well, I found some stuff. I think if asked something interesting, at least, will come to mind before too long.

As someone frequently interviewing people, I ask these kinds of questions because I am interested in what people perceived as a significant challenge in their career, how they tackled it and how they remember it.

THat having said, I would never judge the answer to this question without the context of other questions and answers.

Ditto!!!

I’ve deep dived, investigated, and fixed tonnes of hard problems.

But my memory doesn’t have an index on them. I can’t think of any on the spot and sort them hardest to easiest.

I can think of achievements (and they end up on the resume of course) but those aren’t necessarily debugging a hardest bug, which is a point in time versus months/years of invested effort.

You don’t have to be truthful; I’m sure they’d settle for any complex or interesting issue you’ve had.

Yah, but work long enough in the field and your bound to have a couple doozies. I've been doing software development for close to 30 years now. In that time I've pretty much found every major class of bugs. Processor bugs, yup, compiler bugs, yup, OS bugs (almost weekly these days), library bugs, interpreter bugs, general hardware adapter/etc bugs, you name it. And from all that you would think they are common, but for every "hard" bug I've probably fixed thousands of boring off by one errors, and all the other silly mistakes.

I used to talk about the difficulties of tracking an IO corruption bug in an adapter card and a hardware vendor that wouldn't assist/acknowledge the bug until a another of their much larger customers started reporting similar problems.

But these days, I like to talk about JIT bugs, because I believe anyone using a JIT in production needs to have their head checked. The last one I found literally took me two weeks of writing a bunch of automated tools and burning a crapton of CPU time hunting for a code sequence corrupting a callee saved register in a chunk of code that had a very high reproduce rate but misbehaved after churning away for a few minutes. AKA by the time the problem occurred it was tens of millions of instructions past the actual corruption. I wasn't even sure it was a JIT bug until I found it. Worse it was in a REALLY common piece of code pretty much everyone with a computer has installed, and had been sitting around for a few years before I happened to bump into a pretty consistent reproducer.

Thankfully JIT bugs are rare in most common language runtimes, but what really kicks the JIT problems up a notch is frequently the difficulty of reproducing them and the near inability to do any kind of static analysis outside of execution tracing to find the bugs.

Yeah some of use are just writing enterprise apps in .NET. I found a bug in the .NET framework years ago, but I don't think I can make a big story about it, or remember the exact details, just vaguely. Never had production JIT problems - but then I may not being doing things as close to the bleeding edge or at the same scale as you.

> But these days, I like to talk about JIT bugs, because I believe anyone using a JIT in production needs to have their head checked.

So like every Java, C#, JavaScript developer?

> hunting for a code sequence corrupting a callee saved register in a chunk of code that had a very high reproduce rate but misbehaved after churning away for a few minutes

If you're able to, I usually build projects with a sanitizer to catch strange bugs like these, but the downside is that most runtimes care very little for standards outside of "this seems to work and it's fast enough" so you get a lot of false positives…

true, so I just know a recent annoying one, gdb always stops on sigpipe. Quite annoying when you add a signal handler to ignore sigpipes; yet it keeps exiting... only to realise an hour later that gdb decides that for you on default. Though clear example of RTFM, PEBCaK.

Reply on news.ycombinator.com