Same thing applies to benchmarking, a few guardrails goes a long way. As long as you aren't completely vague ("set up benchmarks", vs. "write tests over a variety of common shapes, be careful to control for environmental variables") the models do quite well. I've checked over the code it's written and it's all fairly reasonable. I do mention cache eviction in the article, this generally wasn't an issue and the model did this automatically on the first pass (back in Opus 4.6 days). I think you might not be giving them enough credit here.
I'm not completely familiar with the writers of this article, but I've never had an LLM try to hack this, even sometimes telling me a target was impossible without quantization or approximation:
"Code based analysis already tells a detailed story, but there is an overlooked parameter: "boundaries of the hardware". " - the myainotez post
This project has spanned several models since Opus 4.5. I think even then such details were trivially known by them, thought they required much more handholding than they do now. I'm honestly not sure how the article writers ran into the hacks they did.
edit: I should mention this article is based on my experience developing an RL and inference framework dealing with trainer-inference mismatch. I don't know if hill-climbing a kernel in an isolated set of benchmarks would perform differently. At the end of the article, I mention what my hill-climber looks like: https://www.nabo.dev/blogs/kernel-primer#section-in-the-age-... It has to go through numerics and correctness checks, then isolated tests for common shapes (where launch parameters are determined by an autotuner, not the agent), followed by end-to-end throughput tests.