Solving Concurrency Bugs Using Schedules and Imagination

It's worth keeping in mind that updates that are not guarded by some kind of barrier are generally not guaranteed to be visible cross-thread / process / server / etc in the order you wrote them. And reorders are rather common in many languages / hardware types / storage systems / log collectors (including stdout because "do thing" and "log that you did it" are evidently not guarded together if there's a race happening), it's not just a theoretical concern.

Generally speaking though: yes, writing it down can help A LOT, and starting with what you can see is one of those obvious-in-retrospect things that are easily forgotten when under pressure. There are often a LOT of possibilities, and getting it out of your head so you can enumerate them more precisely can super duper important. Intuition for problematic sequences to check first will come with time.

Looks a lot like a derivate of a truth table, which is often used to debug multiple input combinations and expected output.

If you happen to be coding in Rust, for really robust concurrency testing, I cannot recommend enough the AWS Shuttle library (https://github.com/awslabs/shuttle) which can find insanely complicated race conditions.

What the Shuttle library is doing is basically automatically going through all the permutations of the schedule diagrams described int his blog post.

We used it at AWS to verify the custom filesystem we wrote to power AWS S3.

If you're curious, I wrote a little tutorial on it here: https://grantslatton.com/shuttle

I insert debug macros with random usleep intervals in critical multithreaded code to expose race conditions. In production they ifdef to nothing.

As someone who’s spent a lot of time with javascript, debugging concurrency issues is second nature.

No better training than a great spaghetti ball of promise chains.