Hacker News Reader

Comment by nsjolj | Hacker News Reader

[flagged]

My previous job was in compiler engineering and I worked on upstream LLVM for a little while. rr was indispensable for me — not just for debugging, but as a tool for understanding how the compiler works in general. For example: you can set a watchpoint on an object in memory, and reverse-execute to the point where it was instantiated. This allowed you to answer questions such as ‘which optimisation pass was responsible for creating this DAG node?’ and such.

Really powerful tooling — I haven’t kept up with it for a while, but last I checked it didn’t work on aarch64/arm64 because of a lack of certain performance counters only available on x86 hardware. The lack of support for other languages (e.g. Go) is also a shame. This has unfortunately rendered it unviable for my day job!

If you’re working in a stack where it’s supported, it’s really quite special.

I believe rr should support other languages exactly as well as gdb does; I've looked at some Rust in it once or twice at least, and IIRC even some Go; though I'm happy enough with doing assembly-level breakpoints/whatnot so I can't comment on more fancy things. Even some OpenJDK JIT! (though there of course source-mapping/stacktraces are non-existent, never mind reading variables, but even then I managed to hack together some basic source-mapping at https://github.com/dzaima/grr#java-jit; though it's rather unusable with rr due to OpenJDK's machine code dumping going directly to stdout, and general slowness).

That said, it would be very neat to have a more complete standard debugging interface across ahead-of-time-compiled code, interpreters, and JITs.

And it does have some aarch64 support by now, though it'll still require a suitable CPU with necessary perf counters.

I do know that RR works for Julia. Not sure why it wouldn't work for Go.

Julia goes through LLVM, which has good DWARF support (and, additionally, until recently, Julia hosted rr's CI, so Julia directly cares about rr). Whereas AFAIU Go does its codegen fully from scratch, without much regard for gdb/DWARF quality.

It says on https://rr-project.org/ that it supports Go programm, what's the status?

rr supports Aarch64 pretty well now.

You might be interested in the rather spectacular upgrade to it, called pernosco :

https://pernos.co/

It’s based on rr ( same author ) but vastly better

There are other commercial reverse debuggers, like Undo.

Pernosco is more than merely a reverse debugger.

rr is awesome. My typical debug workflow:

  * rr record
  * rr replay until crash or exception (set up catchpoint for that)
  * set up hardware watchpoint for borked data that caused the issue
  * work backwards in the data flow through watchpoints and reverse-continue

I would say this fundamentally changed how I approach debugging.

I've used it for years for vulnerability development (when you need to debug why a particular exploitation run went wrong etc.).

It is indeed marvelous for many situations; except for concurrency bugs ;)

It is sometimes useful with concurreny bugs, especially when used in "chaos" mode. Altough it probably masks some classes of concurrency bugs. I wonder if rr + TSAN works.

Why isn't it good for concurrency bugs? rr doesn't record the entire state up to the race condition?

RR supports recording multithreaded programs, but only runs one thread at a time, so it cannot trigger data race bugs.

It can find higher level concurrency bugs though, and has a "chaos mode" that schedules threads in an unfair random manner.

it serializes your problem because you can't atomically snapshot memory when there are multiple threads writing to it

This is true, but for at least some race conditions the rr "chaos mode" can help in tracking them down. Chaos mode basically randomly adjusts its thread scheduling so that it sometimes doesn't run a thread at all for a few seconds, so you run it a bunch of times until it hits the timing window for the race condition; and then when it does you debug the recording. IME it works at least sometimes but it does require that the thing you're testing doesn't take too long to fall over, since you need to do multiple recordings of it.

https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo...

Reply on news.ycombinator.com