The top of the article shows the e2e times on some math RL tasks, with fp8 weights/activations/KV, comparing against SGLang on two models. They use the same concurrent batch size, quantization scheme, and the final lengths were about the same. I think Qwen3 models are probably very mature compared to Muse by now.
30B-A3B MoE
Jam - 67,241
SGLang - 44,696
+50.4%
8B Dense
Jam - 58,821
SGLang - 41,759
+40.9%
Math RL is pure decode, so kernels are the most major factor here.
Also-- I feel as if this has gone a bit off-track. The point of my reply was to comment on agents being actually decent at benchmark development and usage if you tell them to be, not as a comment on how great the harness is (I even state in the article: "it's a fairly simple and minimal setup that seems to work well.").
I don't think it's a footgun at all, and generally feel like (at least OpenAI's models) tend to do well in stating when there is some caveat to watch out for (they'll probably say when they're about to hack for the user to catch). The benchmarking suite they've made is actually very similar to what's in the open kernel challenge. Given that this was initially made many months ago, I'd think Opus and GPT did a pretty good job if someone is betting $100k+ (in prize money) on a similar suite being decent.