Why debugging is all about understanding
futurice.com
futurice.com
Usually breakpoints and stepping through code solves things pretty easily but some bugs can be real stumpers.
Here's some of the tools I tend to use:
1. What has changed? What could have caused this bug to appear?
2. Verify fundamentals: Is it actually running the code I think it does. If I change something that should change something, does it change? If I break something does it break?
3. Verify Input: Make sure it's valid
4. Verify output
5. Timing issues?
6. Slowly remove code "around" bug until it works again
7. Or start back at something you know works and add back code until it breaks.
8. Break out the functionality into an isolated environment so you can test it separately so you'll know if the problem is within the feature or a side-effect of something else
9. If your code makes it too complicated to see and find the bug refactor until it isn't.
10. Go to sleep / Go on a walk. Let your subconscious process it
and finally, lot's of bugs are made self evident by well structured and refactored code, so don't get too busy working on the symptoms rather than the root cause.
2. Are you sure you can you consistently replicate the bug?
I really like Vim, I use it for almost everything, however when I am working on a large C++ codebase I tend to use an IDE that has full support for visual debugging, code navigation and so on. I am always amazed when some more experienced programmers than me spend valuable time grepping files and using GDB on command line.
I don't know about Vim, but Emacs works very well as a visual debugger, for quite a couple of languages. It looks like this for C, for example: http://www.inet.net.nz/~nickrob/gdb-ui.png (it's Emacs debugging itself in this screenshot).
In general both Vim and Emacs can become IDEs that are on par with other offerings. For Python development, I worked with Komodo, PyCharm, Vim and Emacs. With jedi[1] both editors get context sensitive autocompletion (for some strange reason called "IntelliSense" or something by some) and "find definition", "rename identifier" etc. With Rope, they both get nice refactoring support. With Magit/Fugitive, you get the prettiest and most functional GUI for git. With Speedbar/TagBar[2] you get classes outline. With yasnippet/vim-snipmate you get configurable, programmable snippets/templates. And so on and on - all that on top of largely superior editing model that is being developed and improved for 30 years.
I worked with Komodo, PyCharm and a couple of other IDEs, then switched to Vim and I didn't feel that I'm missing something. Then I switched to Emacs because I wanted an editor that I could easily customize as much as I'd like, and VimL just didn't cut it. There is nothing comparable to Emacs in terms of extensibility, LightTable and Atom may get there with time, but I suspect it will take quite a few years. No other IDE even tries to approach this level of extensibility and customizability.
[1] https://github.com/davidhalter/jedi [2] Vim version: https://i.imgur.com/Sf9Ls2r.png
Regarding your GDB screenshot, that is certainly good, I would argue though that using pure text (with some images included) is limiting oneself.
This being said, I really hope that NeoVim will go far enough and bring good enough integration of Node.js and javascript in general (my current work uses these technologies). So far I have been quite disappointed by stuff like tern[1]
[1]: http://ternjs.net
Your code is your debugger. If you will write properly, you will need not to run debugger at all, because code will explain problem to you (or to future user or administrator of your program).
The way your assertion can go wrong is when you don't recognize the errors that your rewritten code will add. At a high level we always envisage perfect code, and existing code is always seemingly full of warts, but no perfect coding plan ever survives contact with reality. Those warts are bug fixes and optimizations which make the thing work. Don't be too quick to rewrite lest you create bigger problems than you started with.
And no, I am not talking about asserts. Assertions are only small part of real debugger.
Typical tools I use in my own programs are: good logging and tracing, which are able to explain what happens inside program without spamming me, some test cases for critical parts and for fixed bugs. Often, I will also split large program or script into smaller independent parts, with their own options, configuration, loggind and test cases. It's all.
For me, when I run out of ideas, "git bisect" is the most efficient debugging tool. On success, it allows me to pin down the exact commit that introduced the bug. It is very fast to do (binary search), and on success it narrows the bug down to a relatively small amount code to check.
Of course, this only works if you have a clean code history: small commits and always a working state after each commit. That is, no intermediate crap commits, and no monster commits doing 3 things at once.
And yes, it requires the type of commit history that I find not many people have the discipline to produce.
Also, bugs aren't only bad - you learn a lot from them too [2].
[1] http://henrikwarne.com/2014/01/01/finding-bugs-debugger-vers...
[2] http://henrikwarne.com/2012/10/21/4-reasons-why-bugs-are-goo...
Over the years, I had less and less use for intense debugging sessions, but I've been reading and writing logs all the time.
Today, I see server side code without logging statements, I immediately ask myself, how do the authors debug this thing?
Related:
> If you want more effective programmers, you will discover that they should not waste their time debugging - they should not introduce the bugs to start with.
That's basically what the quote is implying - you should not be throwing things together and hoping it will work after debugging, but carefully designing to avoid bugs, so there won't be much if any debugging required.
An analogy to this can be found in the aviation industry: it's known that pilots do make errors and planes do crash, but such errors are treated as unacceptable and they try their hardest to minimise them. It's been a very successful strategy.
It is great when these work. Or when you have a high level team that doesn't abuse the system at all and gets things done using these strategies. I have seen too many times where the result is a system that even the creator does not fully understand. Worse, what should be simple changes turn into giant modifications of a Rube Goldberg machine.
Printf debugging forever! ;-)
In my experience, the only way to produce safe multi threaded code is to isolate the threading parts and synchronization and stress test them early on. At this point I use a combination of prints, random delays and simple asserts on invariants that the code should hold. Most issues are reproduced within about 10 seconds of 100% CPU core utilization. Some nasty corner cases may require minutes of grinding away.
Multi threaded debugging is an unsolved problem. There are no universal solutions to this problem. Conventional tools often fail so badly that it is best to write and test your multi threading / synchronization code in isolation, and then applying it to the actual workload which is tested in a single thread.
Another option is trying to build a formally verified model ahead of time (e.g. using Spin - www.spinroot.com) and then write the actual program after the model has been verified.
If you have a race condition in a multi threaded program, a simple printf may be enough to change the timings and make the bug disappear. But a breakpoint and stepping in a debugger is guaranteed to completely change the timings (unless your timings are on the order of minutes).
It's so much easier if you can search what has happened instead of guessing. Also customer experience is much better if you tell them that is fixed instead of asking for details :-).
Disclaimer: I work for log management company (Sumo Logic).
I've seen developers bang away at a problem for days and not be able to describe how it "works" in a code review. They just tried every crazy thing they could think of until it eventually returned results that looked correct.
Don't be that kind of developer. For one, all the other developers are going to make you maintain that code for the rest of your life. Two, never be too proud to ask for help. We can't know how to solve every problem on our own. It's not possible.
There are some who argue against this, and their argument is effectively "if the machine can do it for you, why should you need to know how to do it mentally" --- it fails miserably when debugging, precisely the time when the machine can't do it.
I've heard it phrased thus: "If you don't understand precisely what you need to do, what makes you think you can tell a computer how to do it?"
That's the whole reason why you would want execute the code mentally. If you don't execute the code mentally, you might not realize that your mental model of the third-party component was incomplete to begin with. Mentally executing code is a great way to test your own assumptions.
It also doesn't work if your team isn't co-located.
Ah, if only that were the case. In reality, developers move between jobs, and bad code always becomes someone else's responsibility eventually.
Instead, it's better to understand that as a professional you should be - if not proud of your results - proud of making the best of a bad situation for whoever will inherit your codebase, be that your future self or a colleague.
* You've almost certainly not found a bug in the compiler or libraries or language runtime (somewhat more possible if you're using alpha, beta, or dev versions of something).
* Your CPU, memory, or other hardware are not defective.
* You are not experiencing cosmic rays flipping bits randomly in your data.
* The problem is most likely to be a mistake in your code.
I have worked with programmers who are quick to jump at the most unlikely explanations for bugs, and it's a very timewasting way to work.
This should be in bold type on every page of Stackoverflow.
Some of my "favorite" bugs happened when 2 things were wrong but 1 worked around 2, and then someone fixed one of them.
For searching in space, most programmers have done binary-search with commented code: comment out or bypass half of your codebase. Do that recursively until you nail down where (at which file, which function, which lines) the bug lives.
No, most programmers actually haven't done this for the simple reason that it's highly unlikely to work: most codebases with half of their functionality removed don't actually function. That this kind of lunacy is being asserted as authoritative is galling enough, but it gets worse:
When binary search doesn't work, you need more creative approaches. Consider brainstorming to enumerate even the wildest possibilities: for instance, maybe the bug is in some external resource like a library or a remote service; maybe there is version mismatch of libraries; maybe your tool (e.g. IDE) has a problem; maybe it is bit flipping in the hard disk; maybe the date and time library is sensitive to your computer's settings, etc.
This is advocating debugging by superstition, and it represents toxic thinking. When we have defects in software, we need to be strictly empirical in approach:
1. Make observations.
2. Think of interesting questions.
3. Formulate a hypothesis.
4. Develop a testable prediction from the hypothesis.
5. Gather data to test the prediction.
6. Refine, alter or expand the hypothesis.
7. Go to step 4 until defect has been driven to root cause(s)
If this sounds familiar, it is because it is the scientific method[1] to which we can credit much of modern knowledge and civilization. As to how this specifically relates to debugging, I touched on this in my recent DockerCon presentation[2][3]. Bringing attention to debugging is terrific -- but advocating superstition as a methodology is anathema to true understanding.
[1] https://en.wikipedia.org/wiki/Scientific_method
[2] https://www.youtube.com/watch?v=sYQ8j02wbCY
[3] http://www.slideshare.net/bcantrill/running-aground-debuggin...
Second, when you have a bug and you write a unit test, you are effectively commenting out the entire codebase except for the function under test. When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away.
Sure it's really dumb if you're just excluding a* .c through m* .c (spaces due to HN formatting rules), but figuring out if the problem is in the first or second half of main is not outrageous. I don't think the guy was presenting it as the first step ("If you have no idea where your bug lives"), but I do agree that it comes across as a little naive, since he should have talked about all of the other techniques available first. So the problem isn't so much the lack of a hypothesis, but the inefficient experimental approach of using a brute force technique indiscriminately.
I think the last resort is reached a little sooner for some types of bugs and some experience levels (language, environment, codebase, programming), and yes in many cases it won't even do anything for you.
Personally I always liked dtrace. This guy gave a demo of it at my university once, I thought it was great, one of the best talks I've seen.
> The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search.
In your example, you used this as an assumption, not a hypothesis. And it's not quite right: the assumption you made is that the _presence_ of a small portion of code is responsible for the defect. That's very different than saying that the code is more broadly responsible for the defect. In my experience, very few bugs are caused by the mere presence of some code.
> When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away.
Those are fine solutions for those very specific, very simple problems. Given the problem space, the assumption that the error is caused directly by the presence of some input is well-founded.
main() { a(); b(); }
You seem to be arguing that your example demonstrates the fact that you can localize a bug in the code with binary search; specifically that the bug must exist in either a() or b(), and that running them independently is both possible and will determine the single location of the bug.
This is not true. It is only true in the case that one assumes bugs must have single locations, and that those locations can be found with binary search over what amount to incomplete programs. In other words, you're assuming the prior, begging the question, etc.
It's tempting to say "real code isn't like this contrived example" but in fact this contrived example is the best possible demonstration of why blind binary search is a poor strategy. Let us say the 'bug' exists in a(). You seem to be assuming this is the necessary consequence:
main () { a(); b(); } -- segfaults main () { a(); /* b(); / } -- segfaults main () { / a(); / b(); } -- doesn't segfault
But this isn't the only possibility. With the bug existing in a(), you could also see this result:
main () { a(); b(); } -- segfaults main () { a(); / b(); / } -- segfaults main () { / a(); / b(); } -- segfaults
You would expect this in situations where b() uses data structures created by a().
We may also see this result, still assuming the bug exists in a():
main () { a(); b(); } -- segfaults main () { a(); / b(); / } -- doesn't segfault main () { / a(); / b(); } -- segfaults
If above we learned nothing, here we've actually got a falsehood - our binary search has localized the bug to b(), but it actually exists in a()!
So, in practice, binary search fails to localize a bug in a(). All of these situations can be created by having a() write a global which is relied upon by b() - a() may write to a protected area, b() may have a default value, a() may write nonsense, or pass an integer where b() expects an address - none of this is particularly exotic, they're all the sort of things you get every day when debugging segfaults.
We might now delve into a competing series of contrived examples of a() and b() and argue about their relative prevalence in the world (which none of us are capable of knowing), because if some case is particularly rare, it may make binary search very slightly better than flipping a coin in this case.
Instead, I will point out that this is once again* assuming the prior, and that we have these simple (if shockingly annoying) facts:
1) side effects exist 2) in the presence of side-effects, binary search cannot predict the location of a bug.
And it follows that in the sense that a "scientific" hypothesis has predictive power, then binary search over the codebase is basically the homeopathy of debugging.
Further, there's a misunderstanding about the method. You do a binary search on the "percentage" of code viewed as an instruction stream that is allowed to execute, starting from the beginning. I understand that a can have a different length from b. Given more code, you can also search based on timestamps. You never comment out a and allow b to execute without it, this is nonsense. If that's what you thought I meant, I can understand comments about not actually being able to program. I did not explain this clearly, but only because I had forgotten that it was necessary to explain this clearly, because some people actually don't know how to program, even on HN.
no source,
no debugger,
two function calls
As for the further explanation here, allowing a debugger makes the search irrelevant - there's only one piece of information the binary search can tell you: which function causes the segfault, but as you observe, that information's available from the debugger already. And no, it doesn't matter if you reframe things as "a percentage of an instruction stream".
The objection to binary search is not "binary search in the codebase can never tell you anything," it's that it's a method that can't -- not "doesn't" but "can't" -- produce a non-trivial testable hypothesis. Does this mean you can't ever use it? You may have no choice. But it's a terrible place to start, and most of the information it offers is available through other means, often with more precision.
It's not that this approach motivates root-causing failures (though that's true). It's that uncaught exceptions are programmer errors, and it's tautologically impossible to correctly deal with them. Attempting to do so can make things much, much worse.
To make this concrete: I've dealt more than one production outage caused by the mere presence of an uncaughtException handler. If the program had merely crashed, a hundred requests may have been interrupted, but the program would have restarted and resumed servicing requests. Instead, the exception was thrown from a code path that had a database transaction open with database locks held. Because the uncaughtException handler just logged the exception and otherwise ignored it, that transaction stayed open (and the locks remained held) until a human got into the loop -- interfering with tens of thousands of subsequent requests. That's much, much worse. If the process had just exited, the db connection would have been closed, the transaction aborted, and the locks released.
An unexpected error handler can't know the complete state that the program was in because by definition this wasn't a state that the programmer thought about.
If it interests you look at Erlang's OTP, it solved this problem years ago and works very well with that.
The former is true even if you are doing exploratory development: in that case, you need to understand what has been found so far and what the next iteration is intended to investigate.
If often found some bugs in a reporting software, that showed the people wrong numbers for years.
Or if something needs to be re-implemented, say for performance reasons, you get the algorithms and notice that the old version wasn't only slower but also wrong.
When this happens, I start to question if the people really "use" the software in the first place. Or if they just get some wrong sens of safety from it, which calms them, but the underlying system is mostly random and they could simply make up their own numbers.
This is a great page (with a video) about what I mean
Fault/Error/Failure provides a better language for talking about systems and their processes. It provides a diversity of solutions from changing the source code, to handling an error, to just letting it crash...the last is impossible to cast as a solution in the nomenclature of bugs.
There are times in which there the ambiguity of "bug" matches the necessary ambiguity of the context...just as there are times when describing the sun moving across the sky is useful. But the geocentric cosmological model breaks down when we're interested in predicting the location of Jupiter in the night sky.
That can, as a very conservative estimate, cut down time spent debugging by 50%.
Life is too short to spend it debugging undefined behaviour.
The second step is using a good tool that helps you make sense of those logs, and includes good debugging information with the logs. A tool like Rollbar (disclaimer: I almost work for Rollbar) makes it super easy to analyze patterns in your errors and logging, find out who experienced the error, and to hear about the bugs before your customer, who may have become used to them.
Anecdote from my previous employer: we had a terrible piece of legacy software that regularly had modal pop-ups warning of errors this or that that doubled the amount of time our clients took to do stuff. They were so used to it that instead of reporting the error, they just dismissed them.
Worst times are when you jumping around the flow area.