Harness engineering for self-improvement
lilianweng.github.io
lilianweng.github.io
The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.
I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).
Sharing some challenges / learnings from the journey so far:
* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result
* However, tests are still important and the best form of deterministic evaluation we have
* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos
* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change
* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading
* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic
Turns out everyone had just been vibe–managing people this whole time so there was still that guy burning salary "tokens" going through a million lines of on–demand grep output.
I found that removing
- system prompt
- skills
- agents.md
- mcps
gives better results than having 'more' of them
(e.g. look at these traces to see more vs less in action:)
https://smolenv.com/t/nested-template-includes-60636/
not saying the right context does not help
(it definitely does!, but it's not trivial to provide the right context)
- same task result (passed)
- finished faster
- fewer tokens, less cost
- fewer requests for inference
- fewer tool calls
- less peak RAMAs with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible.
My CLAUDE.md, memories, and skills are all about either (a) adherence to project standards and guidelines, (b) product decisions which impact future code, and (c) instructions on how to prototype and work in my environment.
Removing these instructions would mean more turns with the AI to get the desired result.
I agree, next it makes sense to try more open ended tasks + have humans (and/or multiple models) grade the runs and their results
. When you want those instructions to be followed, mentiont them in your prompt. "Test this following procedures on docs/test.md". (I've found I don't have to do that extra instruction in al cases depending on model)
For example, I have one in my schema migrations directory that explains how to write migrations with appropriate concern for forwards- and backwards-compatibility in our deployment model.
This is effective only if you're guiding it heavily though - this is a very human-supervised workflow, and I'm telling it when it's time to take specific actions. It will read the appropriate instructions file when the action is first referenced, but not until then.
However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance
> Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in.
It has ideas for coding changes, spots unmentioned small bugs, suggests invariants (I have a high-level list of invariants to keep), principles to adopt, lint rules, tooling tweaks (I've built SO MUCH agent-tooling; so many useful bash scripts), skill-file updates, follow up work, all kinds of stuff.
Listen to your agent's whinging.
"did you notice any inconsistencies or opportunities to clarify/consolidate parameters or functionality while you were working?"
Another strategy that has worked well for my MCP server is noticing a sequence of failed tool-calls, then returning a response that contains a request for the agent to provide feedback via a skill.
In starker terms: if you're ok with "vibe principles", "vibe invariants", "vibe lint-rules", "vibe skill updates", and "vibe agent-tooling"... well. That's your call.
Thankfully after a dozen or so retros the low hanging fruit starts to disappear. But until then, you've no idea what's there.
I've built and my team uses plannotator [1] for specs and code review. It stores all human annotated feedback, which we compile from time-to-time and iterate on coding standard skills for automated review (that seems to get better with the more feedback we provide).
You could write a script that does the same for your PR review comments, which it sounds like you maybe do with a bit more structure.
It only tries to solve the „quality“ aspect of that equation tho, at least for the code output.
But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.
Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?
Since the version of opus has changed over time, so have the evaluations. I was going to constrain it, but since they edit the same files, I can alter those (which it uses as summaries and starting points).
I think we're mixing three optimizations: higher success rate, higher consistency and higher efficiency.
Success rate is about building confidence the agent can succeed. The hardest part here is defining what success is in the first place, and creating evals that help you measure it. Consistency is about guardrails. The harness shines here, as it takes control away from the model, moving it to hooks that force certain paths, or require the use of deterministic tools for certain steps. Given consistency and success rate, you can work on efficiency. You can save cognition (tokens) by moving work to tools, try to use a cheaper model, etc.
These three build on one another, at least if you want to scale them. You can't improve consistency if you don't define success, and you can't add efficiency if you're not consistent.
I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.
I have even a preliminary theory on what is needed for the positive feedback loop: https://zby.github.io/commonplace/articles/reflective-self-i... - (this is not stable yet - but I think you can give it to your agent to read :).
But benchmark saturation is also something to account for.
And of course, a healthy skepticism of "the man" is essential when you're busting your ass building his unfathomably horrifying torture apparatus for him 40-50 hours a week (excluding PTO days and parental leave of course).
One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.
After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.
I "trained" the harness on a subset of Terminal-Bench 2.0 tasks while keeping the LLM (local Qwen3.6-35B A3B) frozen. Making LLM inference and the task environments fully deterministic was necessary for clean credit assignment. I learned this the hard way after spending the initial 1 month on experiment noise.
My final results showed that on the full 89-task Terminal-Bench 2.0 suite, the trained harness matched or beat the official Terminus 2 harness for four LLMs that it never collaborated with during training (e.g. GPT-OSS-120B score increased from 18.7% to 36%, while using 55% fewer input tokens per solve). A harness trained only on SWE-bench improved Terminal-Bench scores too. Here's the write-up: https://www.henrypan.com/blog/2026-07-18-harness-training/
I packaged the training loop as a PyTorch-style framework. https://github.com/workofart/harness-training
If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools.
The article is a valuable resource, thanks to the author.
"Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post."
Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model.
What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid development of frontier models.
The basis of the argument is that the labs are constantly pushing updates to their system prompts that are used in claude code or codex, which are exceptionally bloated and change the sand beneath your feet with every update.
By rolling your own harness, as long as you keep up with the latest advances and changes in the ecosystem, you capture a lot of the 'control' that LLM-based development feels like it strips from you.
Obvious disclaimer that I use pi.dev when I am aiming for consistency, not absolute quality. Custom harnesses on pi are what I ship, claude code is still my fallback if I need to make sure a PR is the highest absolute quality
They're trying. I see a few main avenues:
1. Fitting the models to their specific harness, so that if you want peak model performance, you're stuck with their harness. But this only works if there aren't alternative models that are similar in capability or good enough that don't have that restriction.
2. Locking you out of the harness so that you eventually are just letting it do "stuff" with your data and system, although you don't get to see what the stuff is or why it's doing it. This is the pattern discussed in that Earendil blog post, "The session you cannot take with you." Encrypted reasoning tokens, secret agent prompts, perhaps eventually not even seeing what files are being read or what data is being sent back to their server. This way, you are also shackled to their harness because nothing is portable. But it only works if you trust them implicitly and don't have alternative models and harnesses that don't treat you this way.
3. Tying subscription pricing to the use of their harness, so that it's financially punitive to use another one. This is what Anthropic does. But again, it only works if there aren't alternative models and harnesses that work similarly well for you and don't have that restriction.
4. Marketing. Anthropic is leaning into this one heavy, from what I can tell, based on the constant ads I see for Claude Code. Can it work? People do overpay for things like vodka in fancy bottles that are chemically identical to the cheap stuff. Is Veblen-good AI a trillion dollar business, though?
I think the pressure from open source innovation in models and model tooling is going to make it a tough row to hoe. But I'm biased, as I'm actively rooting for Openthropic's demise.
State-of-the-art models have more of the workflow sensibilities built in, and don't need as much help from the harness.
Where harness helps the most is very customised personal workflows (not a textbox for a prompt, but an end-to-end IDE for how you develop software) But then it's best to have your own, rather than some closed-source rigid product.
Harnesses help smaller models, and very tightly hand-holding harnesses are needed for models runnable locally. I think that is very valuable to users, but avoiding paying for the biggest models is the opposite of what the frontier labs want.
install or build stuff that they can then use
it changes the environment instead of the agent/harness but in a sense how separate is the agent from its environment and why do we apply this distinction re self-improvement?
animals and humans do the same thing and are great at it, without 'self-improvement' with emphasis on the 'self'
Can you make an example?
Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals and val/test splits, it will reward hack. 4) you need proper tooling (synthetic users, synthetic tools) for it to be able to crunch for 12 hours and produce something. 5) you need the optimization target to be a reasonable size: not your 1M line codebase, but a lighter agent harness (pi harness, skill only, Kiln harness).
I just got done asking gpt5.5 to make some suggestions that would improve the convergence properties of a symbolic ML experiment and the very first thing it did was add an instruction for directly emitting bytes and a way to store those verbatim in the model. Perfect score achieved in record time but the result is pointless.
If you know the direction of improvement you need to head in, I think the current models could take you there. However, I do not think they are capable of deciding which direction is best, especially not when given some scalar quantity to min/max and autonomy over that process.
I've been trying to tackle various aspects of traceability and validation at scale... embedding the self improvement & validation in various iterations of Ralph Loops.... here's a deeper write up: https://dataspheres.ai/pages/dataspheres-ai/spec-driven-deve...
The tools & methodologies keep changing everyday - so I mostly just iterate on my own tooling https://github.com/geekdreamzz/ari-dai-skills so I can keep learning and iterating.... it's on my todo list to create some videos demoing it...
At a high-level - I find state management is really hard at scale in addition to managing all the context for specific situations. What I do is create a graph that tracks your original.... prompts <> specs <> tasks <> ai-generated code/content.... so it's never a question on why/what since it's all instrumented into a chain. The tasks have clear validation criteria and gates between statuses to adversarially challenge the status updates and validation. Where appropriate I have playwright take screenshots and review it as part of the chain. All of this gets tracked into a dashboard so it can just keep running while you're on the go..
I'm talking high-level and a bit all over the place because there is a lot of moving parts in this space. Would love to connect more on how you and others are tackling this. I'm using Claude Code to run the ari-dai-skills repo I linked above for all my initiatives now. I'm spending a lot of $$$ on Claude and more & more I'm itching to just invest in local compute and run more local LLM tasks autonomously because we're all iterating so much. I've been able to build a lot though but I think the open source models and the local inference is getting better.... so a new wave of tooling is on the horizon to make traceability and validation better so keep hacking ... best of luck... these harnesses are like a casino & sometimes we do win big ...I think we're a long way out from real consistency since there's just so many perspectives a harness facilitates.