Another Case of Obscure CPU Nondeterminism
robert.ocallahan.org
robert.ocallahan.org
rr is a lightweight tool for recording and replaying execution of applications (trees of processes and threads).
This is a huge deal. While I prefer the Cilk approach (automated debugging pinpointing places which can theoretically execute non-deterministically), it's not always applicable and isn't always or even often applied where applicable. This is definitely the next best thing, and in absolute terms, it's pretty damn good.
rr's pretty great for a lot of purposes though -- I think the clearest use case for it is the "memory corruption happens but doesn't get noticed until execution has progressed a long way forward from the actual site of the bug" kind of problem, which it can speed up debugging of massively.
If you figure out the cause of a bug that rr chaos mode was not able to reproduce, please file an rr issue and explain the situation. We've generally been able to improve chaos mode to find hitherto non-reproducible bugs.
That's not actually the hard part :)
It honestly really wouldn't help that much, even if it wouldn't be infeasible for other reasons (increased resource usage, even of their cited 1.2x, is a ton, etc).
Even if you could completely reliably restore the state to a random other machine than the one it ran on (IE possibly different arch/memory/etc configuration), and you met all the requirements, the main part rr helps you with is reproducing the failure, which, most of the time, google could actually do anyway.
The hard part is figuring out what went wrong. Remember Google already has asan, ubsan, and tons of other stuff running. So the trivial causes are pretty much not occurring. It's not like people look at the flakes and say, welp, i messed up this one variable and that was that!
It's usually torturous debugging of trying to understand the set of conditions that have occurred.
IE The reason there are so many flakes is because there's so much that can go wrong. Not because people can't reproduce the failures.
Also note that in Google's world, RR would probably not be compatible with how the tests are run anyway (RR wants fairly exclusive to the perf counters, but the tests may be getting sampled), and for a set of flakes, perturbing the performance counter settings will change things enough to make them less flaky!
I've developed automated debugging tools, I debugged hundreds and hundreds of rare, irreproducible bugs or just bugs that are hard to figure out, I found things like memory ordering bugs in others' concurrent code just by eyeballing it because the bug was too rare to wait for it to reproduce, I debugged concurrent code on multicore chips without memory coherence, on faulty hardware, etc. You're entitled to your own opinion about what "the hard part" is, but I'm likewise entitled to mine, and I think rr is awesome and I disagree with you on every point.
So not only are you able to reproduce the error, but you're also able to go backwards to find how it happened!
I, for one, barely use gdb anymore because reverse debugging with rr makes debugging so much easier (well, technically, I still do use it, since rr is not an entirely new debugger, you still end up in with a gdb prompt).
FWIW, one of the first things I tried when I first used rr was to debug a crash I had debugged years earlier, that was due to a miscompilation by GCC. As that had happened years earlier, I didn't remember the details of what code was miscompiled in what particular way, but I do know that debugging that took a long time, and I had only figured it out by chance because valgrind pinpointed to related code. With rr it only took minutes to find the root of the crash.
For some (and i'd guess a bunch of the people i mentioned in the parent comment), they definitely find this easier, but just to present a contrarian position: i actually don't. I admit to being weird - in a former life, i was a gdb maintainer.
I also was trained by people who believed the right approach was not to immediately try to find the sets of conditions and variables that caused your problem and declare victory, but to go and meditate upon the code and think about it until you understood it well enough to understand why this might happen even when you think it couldn't. That will often enable you to understand the code well enough to see what else is wrong.
(Again, i don't claim it's better, i just claim that's why i tend not to care about RR. The hard part for me is the thinking about the code, not the finding the sets of conditions and variables that caused a particular set of errors)
It's definitely the case that, personally, when i follow the "find conditions, fix bugs" approach, i tend to write much buggier code (even with good testing strategies) than when i follow the other way.
I think eschewing debugging is fine for code you understand pretty well and when you already have significant information about the failure. But when those conditions aren't met, debuggers are very useful. (NB, if you use logging code and think "I'm not using a debugger!", you're just using a bad one.)
It's true that a good debugger tempts one to think less deeply about the code than one should, but that temptation can be overcome.
The Visual Studio feature you mention is probably Intellitrace, which is considerably more limited than rr and doesn't handle heap state.
Maybe you can reproduce the failure once in a while, but can you reproduce it on every run, so that when you apply the debugger to a particular run you're sure to see the failure? That's what rr gives you. Furthermore rr lets you debug the same execution over and over again, so stuff like event ordering and object addresses stay the same between debugging sessions.
And as glandium said, on top of that we build reverse execution, which is a real game-changer. Debugging is about tracing effects back to causes, and reverse-execution is what you actually want for that.
FWIW rr has a fair number of users who find it really does help a lot for all kinds of debugging tasks, not just those involving flaky tests. Some of those users are even at Google :-).
No, it can get by with just one of the general-purpose counters, though two is preferable. Most modern Intel CPUs have at least four GP counters, plus some other dedicated counters for the most commonly used events (instructions, cycles).
> Also note that in Google's world, RR would probably not be compatible with how the tests are run anyway
That may well be true, but I suspect it's more likely to be some other issue, e.g. that the tests run in VMs that don't virtualize the PMU.
It's true that rr perturbs tests, making some bugs difficult to reproduce and making other bugs show up. However, a single failing run captured by rr is almost always enough to figure out the bug. We also have some techniques for randomizing perturbation (e.g. of scheduling) to expose a wider variety of bugs.
Maybe it's the kind of thing that is hard to get until you try it. I'm like glandium -- I almost never use gdb proper anymore, and hard bugs seem to vaporize as soon as I run rr replay. In particular, the idiom of "notice a bad value, set a watchpoint, reverse-continue" completely changed my relationship with my debugger.
For me, debugging flakes is a small part of the value, although maybe that just reflects the kinds of systems I work on.
I do agree there's probably no need to run all tests under rr, at least with the setup we have at Google.
I just learned about the ss utility last week (for finding out what application is using a particular socket, as an alternative to lsof or netstat), and now I've just learned about rr.
I'm curious about that mention of ARM --- since I always thought it was a far simpler and more rigidly defined architecture with fewer edge-cases/undefined (more like implementation-defined) behaviour than x86.
This disappoints me, as I'd love to see rr ported to PowerPC, which is another LL/SC architecture that I suspect might run into this issue.