I'll update the post in the coming week with a section on benchmarking. The current post does have some details on measurement, but I agree that maybe it's not covered in the required depth.
Primer: https://ornith.ai/defense_kernel_hack.html
In-depth(follow links etc): https://x.com/myainotez/article/2037258418843771349
Parts relevant: https://www.coreauto.com/blog/when-ai-starts-writing-systems...
----
When you do this with benchmarking, the same thing happens but on a much deeper, shittier, harder to understand level. It will set up the caches to benefit your kernels, it will set up the timing to benefit your kernels, it will set up the evaluation order to benefit your kernels. I've seen half-cache flushes, which only benefit large shapes.
I wouldn't be surprised if you pointed a language model on your benchmarking with the comment above, and it found something that is reward-hacking your task.
Same thing applies to benchmarking, a few guardrails goes a long way. As long as you aren't completely vague ("set up benchmarks", vs. "write tests over a variety of common shapes, be careful to control for environmental variables") the models do quite well. I've checked over the code it's written and it's all fairly reasonable. I do mention cache eviction in the article, this generally wasn't an issue and the model did this automatically on the first pass (back in Opus 4.6 days). I think you might not be giving them enough credit here.
I'm not completely familiar with the writers of this article, but I've never had an LLM try to hack this, even sometimes telling me a target was impossible without quantization or approximation:
"Code based analysis already tells a detailed story, but there is an overlooked parameter: "boundaries of the hardware". " - the myainotez post
This project has spanned several models since Opus 4.5. I think even then such details were trivially known by them, thought they required much more handholding than they do now. I'm honestly not sure how the article writers ran into the hacks they did.
edit: I should mention this article is based on my experience developing an RL and inference framework dealing with trainer-inference mismatch. I don't know if hill-climbing a kernel in an isolated set of benchmarks would perform differently. At the end of the article, I mention what my hill-climber looks like: https://www.nabo.dev/blogs/kernel-primer#section-in-the-age-... It has to go through numerics and correctness checks, then isolated tests for common shapes (where launch parameters are determined by an autotuner, not the agent), followed by end-to-end throughput tests.
Look at this challenge:
https://openkernelchallenge.dev/#leaderboard
If you have the tokens, let your harness go at let's say the first bf16 problem, I am curious how far you will get with it (1.73x is the leader and 2.04x is roofline). I think if you manage to get above 1.63x, I would be really interested in learning how you did that.
The top of the article shows the e2e times on some math RL tasks, with fp8 weights/activations/KV, comparing against SGLang on two models. They use the same concurrent batch size, quantization scheme, and the final lengths were about the same. I think Qwen3 models are probably very mature compared to Muse by now.
30B-A3B MoE
Jam - 67,241
SGLang - 44,696
+50.4%
8B Dense
Jam - 58,821
SGLang - 41,759
+40.9%
Math RL is pure decode, so kernels are the most major factor here.
Also-- I feel as if this has gone a bit off-track. The point of my reply was to comment on agents being actually decent at benchmark development and usage if you tell them to be, not as a comment on how great the harness is (I even state in the article: "it's a fairly simple and minimal setup that seems to work well.").
I don't think it's a footgun at all, and generally feel like (at least OpenAI's models) tend to do well in stating when there is some caveat to watch out for (they'll probably say when they're about to hack for the user to catch). The benchmarking suite they've made is actually very similar to what's in the open kernel challenge. Given that this was initially made many months ago, I'd think Opus and GPT did a pretty good job if someone is betting $100k+ (in prize money) on a similar suite being decent.
Thanks for bringing up these points, it's definitely good food for thought to add to the blog!