cgaebel

Thanks for working with Jespen. Being willing to subject your product to their testing is a huge boon for Redpanda's credibility.

I have two questions:

1. How surprising were the bugs that Jepsen found?

2. Besides the obvious regression tests for bugs that Jepsen found, how did this report change Redpanda's overall approach to testing? Were there classes of tests missing?

We are similarly sad about how unavailable Intel PT is in VMs. In 2022, being unavailable on Macs and VMs raises the barrier to entry extraordinarily high for many people in our target audience. Not sure if working outside of work is your cup of tea, but we've found Intel NUCs to be <$1000 and an unobtrusive way to play with these features at home.

Good point about overhead. I've moved the 2%-10% number front and center, and wrote up a bit more detail about where that comes from in a new wiki page: https://github.com/janestreet/magic-trace/wiki/Overhead

We'll think about adding flame graphs. We unfortunately have little experience writing responsive web UIs, the excellent Perfetto developers did all of the heavy lifting on that front. But who knows, maybe an enterprising Open Source Contributor could help us out. I see Matt Godbolt was asking questions in their discord the other day...

You basically understand how this works; you see everything, but there might be gaps in the trace. In our experience they're rare (< 10 per multi-millisecond trace), short lived, and magic-trace can mostly infer what happened in that period fairly easily. You'll see these show up in the final trace as a little arrow that says "Decode error: Overflow packet" when you click on it, and the trace might look a little wonky (hopefully not too wonky!) from that point on.

In fact, if you look carefully at the demo gifs in the README, that trace had 5 decode errors! Nonetheless, it was extremely usable.

Snapshot sizes are configurable--you can go back as far as you like. However, the trace viewer tends to crash when the trace files reach the hundreds of MB and you'll need to do some work to set up a trace processor outside of your browser for the UI to connect to. The UI will offer up some docs if you actually run into this.

I'm so glad you asked us about PMU events, we've been thinking a lot about those. These are available in traces of the efficiency cores of Alder Lake CPUs, but nothing else. When we get our hands on a server class part with PMU tracing we'll add support ASAP. We conjecture that it will be absurdly useful to see cache events on a timeline next to call stacks.

Thank you for this. I've made the change, but it looks like it may be several days before github gets around to refreshing the language statistics.

Broadwell works, Skylake or later works better. We go into more detail about what platforms we support and why in https://github.com/janestreet/magic-trace/wiki/Supported-pla...

To be clear: bitcharmer says "we" to mean "fellow HFTs" not "Jane Street".

It can! You can read more about how to do that yourself here: https://perf.wiki.kernel.org/index.php/Perf_tools_support_fo....

magic-trace uses perf. If you want, you can think of it as a mere "alternative frontend" for the Intel PT decoding offered by perf.

Hi HN! I'm Clark, one of the maintainers of magic-trace.

magic-trace was submitted before, our first announcement was this blog post: https://blog.janestreet.com/magic-trace/.

Since then, we've worked hard at making magic-trace more accessible to outside users. We've heard stories of people thinking this was cool before but being unable to even find a download link.

I'm posting this here because we just released "version 1.0", which is the first version that we think is sufficiently user-friendly for it to be worth your time to experiment with.

And uhh... sorry in advance if you run into hiccups despite our best efforts. Going from dozens of internal users to anyone and everyone is bound to discover new corners we haven't considered yet. Let us know if you run into trouble!

Hi! We've just released a new version of magic-trace. Happy to answer questions here.

It's happened! We have some internal magic that lets us run ml directly: ./my_bash_script_replacement.ml, and it makes replacements fairly painless.

But we don't do it often. Each language has its strengths and weaknesses, and the constraints of a small script tend not to change that much over time. The script either starts other executables and pipes some text around, or it does some computation. The choice of which to use is often (but not always) clear.

You must not live in San Francisco.