Hacker News Reader

Comment by saagarjha | Hacker News Reader

Don't know much about Julia, but it seems like this debugger works by running the code in an interpreter? Is there tooling around debugging compiled Julia code as well, or is this the only way to feasibly debug Julia? It any of this tooling standardized so it could work in say GDB or LLDB (taking advantage of their command syntax, Python API, etc.)?

View on HN

Even newer... so early days, but you might also be intereting `MagneticReadHead.jl`. The debugging instrumentation gets compiled into the code.

https://github.com/oxinabox/MagneticReadHead.jl

Author of MagneticReadHead.jl here. Hit me up with any questions.

It is a a bit less polished than Debugger.jl right now.

Iirc the first attempt, Gallium, did that. I believe it was technically super ambitious and a bit brittle maybe. It worked with Julia version 0.5. Maybe one time Keno or someone else very capable has time to revive it, I don't know.

I wouldn't be surprised if it wasn't so much "brittle" as that Julia internals were too much of a moving target before reaching 1.0 to make it worth the effort of keeping it working between releases.

It was a bit of both but by far the bigger issue was that compilers emit terrible debug info. It's barely good enough to debug C/C++ code and for Julia code it was impossible to reliably get values of local variables and set breakpoints.

This effort is effectively the revival of half of Gallium. At the end Gallium was a combination of two debugging techniques — it was both a "native" gdb/lldb-style compiled debugger as well as an interpreter. The native debugging was fast but indeed challenging and rather limited — most Julia code gets optimized away when it gets compiled leaving you with a rather unhelpful stack frame that's challenging to line back up with your original code. So there was also a rudimentary interpreter (the ancestor of this interpreter) for other cases.

Is there no way to compile Julia with debugging symbols and frame pointers?

Something like this is currently being attempted with MagneticReadHead.jl[1] which also just saw its first beta release. It's a pretty cool tool, using Cassette to essentially do an extra compiler pass, descending into code from other modules and inject the appropriate debugging tools.

[1] https://github.com/oxinabox/MagneticReadHead.jl

MagneticReadHead is a bit different than what I think your parent post was asking about—that's more what Gallium was. The breakdown of approach is:

1. Gallium was a traditional gdb/lldb-style debugger

2. Debugger.jl is a pure interpreting debugger

3. MagneticReadHead works by compiling a modified version of the code you want to debug that is instrumented to add debugging hooks (this is different from how gdb/lldb work).

Oh I misread the parent. My bad.

It does by default so you can use gdb/lldb and perf on Julia code.

If you want to debug Julia itself, you can use gdb/lldb on it. That plus using the `jl_` function to dump values gets you a pretty decent debugger if you're comfortable in gdb/lldb, but it's not an experience that most Julia users would be happy with and you cannot evaluate code in the current stack frame's context or set breakpoints with Julia expressions as conditions. Debugger.jl lets you do all of that.

Yes, the original attempt was a bit brittle. So this attempt addresses things from the other end and having something robust but slow. From here on, the plan is to keep adding features from Gallium that improve performance.

Yup, Gallium used DWARF debug info like gdb/lldb and could debug optimized, compiled Julia code. It's named Gallium because the DWARF standard constant for the Julia language is 31 and gallium is the 31st element in the periodic table :) Unfortunately, compilers generate horrible debug info and it was very hard to get even basic stuff like knowing the values of all in-scope local variables to work reliably and making breakpointing work was very much a work in progress.

Debugger.jl on the other hand is slow, being interpreter-based, but very reliable. The down side is that it's really slow, so if you need to debug performance intensive code, you might be in trouble. In the future, the plan is to steal tech from Gallium [1] and MagneticReadHead [2] and use them to create a hybrid interpreted/compiled debugger that is as reliable as Debugger.jl is now but which can run code fast enough to debug even compute-intensive workloads effectively.

[1] https://github.com/JuliaDebug/Gallium.jl

[2] https://github.com/oxinabox/MagneticReadHead.jl

The history here is a bit as follows. Back about four years ago, I set out to write a debugger for Julia and while I was at it, I figured I might as well fix all of the other things that have always bothered me about debugging. As an example of that, it had the following features:

  - Being completely written from scratch in Julia
  - Mixed debugging between Julia and C/C++ code
  - Integration with https://github.com/mozilla/rr for time traveling debugging
  - Debugging windows programs running on a linux host under wine (and in particular even as an rr replay).
  - Zero overhead debugging by using mixed modes (DWARF based debugging generally, a much more accurate interpreter based one when possible).
  - Very precise location tracking, on an expression basis, not just a line-by-line basis
  - Multi-process debugging by riding along RPC calls, even across machine and even as separate rr recordings across machines

As you can tell from this feature list, that project was a bit over ambitious and in the end failed to be a particularly useful debugger for Julia. It was useable for a bunch of basic cases, but fairly brittle. Also, then 0.6 came around and all of Julia's internal APIs changed and porting over the entirety of Gallium would have been quite hard.

After 0.6 was released, I decided this wasn't working out for a Julia debugging story and pulled out just the interpreter and UI parts from Gallium (that package was called ASTInterpreter2.jl). This worked ok, but was lacking a bunch of features that I never got around to implementing, because the lead up to 1.0 required a whole bunch of compiler work that nobody else was available to do. This work now is based on that work from 0.6, but actually making it work nicely and paying a whole lot of attention to it that I was never able to. I'm very happy that Tim, Kristoffer and Sebastian decided to take this on, because I feel like my workload has only increased over the past year ;).

I still very much want all the ambitious features from above, and we do still have the code (and much of it is actually in use, just not for debugging) and I'm fully expecting them to make a recurrence. However, this time around, we'll start with a stable, usable system and gradually make it faster and add features, rather than building an unstable system with all the features and trying to fix it. The former approach is much better for getting usable tools out (though sometimes the latter can lead to more interesting results in a research setting). The other thing to note is that Julia is much more stable now, so it's easier to keep things running without breaking every few months.

The history of this makes me think of Gall's Law [1] which was on HN just the other day. It will be truly awesome when we do get all of these features you originally wanted into a standard, stable, usable Julia debugger—and I have no doubt that we will eventually :D

[1] https://news.ycombinator.com/item?id=19430948

I don’t know enough to answer your question about hooking into gdb/etc, but are you familiar with with the code_lowered macros? They will output a given function/scope in whatever lowered form you request (lowered Julia, LLVM, and something else).

Also there is currently early plans for Debugger.jl to incorperate compilation at some point.

Reply on news.ycombinator.com