Designing a Self-Improving Agent Harness: How to Tell Real Gains from Noise

Part 1 of the Self-Improving Harness series. Over five days I built an optimizer agent that edits the harness around my production diagnosis agent, ran it for hours at a time, and got it to a result that held up on repos it had never seen. The evolved harness now runs in production. This post covers the design and why I made each decision.

---

Why I built this

I had been reading research papers and blog posts about self-improving agents, where one agent edits the prompts and tools of another agent and keeps the changes that help. I wanted to learn how to actually build one, on a real agent I run in production, not a toy.

I also had a practical reason. I wanted my diagnosis agent’s harness to be more efficient: find the right file more often, in fewer turns, for less money. Tuning it by hand meant changing a prompt, running some cases and guessing whether it helped. I wanted something that could propose changes, measure them properly and only keep the ones that really helped.

I didn’t have unlimited time or budget. So I wasn’t trying to answer an open research question. I wanted to build the thing properly, using what the papers had already found matters, and see if it worked on my agent.

What “harness” means here

DiagnosisAgent is the part of my incident pipeline that reads a production error and works out which file is broken. It explores the repository with a few tools (read a file, grep, verify a symbol exists, find a function’s callers), then submits a diagnosis through a grounding gate that rejects anything not backed by code it actually read.

The model does the reasoning. Everything around the model is the harness:

• the task prompt that describes how to investigate
• the tool descriptions that tell the model what each tool does
• settings like the turn budget (15) and how much of a file one read returns (12,000 characters)
• control flow, like what happens when a run ends without an answer

Harness changes are cheap to make and easy to ship, and a lot of day-to-day agent quality comes from them. So the question I wanted to answer was whether an agent could make those changes for me.

The first step was making the harness editable. It lived in Python strings, so I moved it into files: a prompt template, a JSON file of tool descriptions, and a settings.json. I checked that the new prompts were byte-identical to the old ones. After that, the optimizer had a clear, limited set of things it could change, and every change was a readable diff.

What the papers told me

Four papers shaped the design more than anything I built.

GEPA showed how to propose changes: have an LLM read execution traces, write down what went wrong, and use that to suggest a targeted prompt edit. It’s cheap and it works, so my proposer is GEPA-style.

RRSI (Google, Regularized Recursive Self-Improvement) showed what goes wrong. Without regularization, the loop memorizes the tasks it’s scored on. In their experiments an unregularized evolved harness scored 92.8 on the tasks it evolved on and 40.3 on held-out tasks, and the unevolved harness scored 39.7 on held-out. So almost all of the “improvement” was memorization. Their fixes became my acceptance rules:

• a noise floor measured on the unchanged harness
• cost-aware acceptance
• a leakage critic that reviews every edit before it’s evaluated
• an edit history the proposer can see

AI2 / UW (Rethinking the Evaluation of Harness Evolution) asked a simple question: with the same budget, does harness evolution beat just running the agent more times and taking the best result? Often it doesn’t. The gains shrink or disappear once you compare against reruns and test on tasks the optimizer never saw.

OpsHarness is the closest thing to what I was building: a self-evolving harness for production root-cause analysis, used in industry. So a self-evolving production diagnosis harness isn’t new, and I’m not claiming it is. What I didn’t find, in the papers I checked, was one that combined RRSI-style regularization with AI2’s rerun comparison, on held-out repos, with repeated trials. I wasn’t aiming for that gap, but following all four papers meant my design ended up covering it.

Measure the noise first

Before building the optimizer, I re-ran the unchanged agent on the same benchmark cases. 16–23% of the cases it had passed before failed the second time, with no code changes. Across repeated runs, about a quarter of cases flipped between pass and fail.

That changed how I thought about everything else. If you test an edit once and see +3 points, you haven’t learned anything, because the same harness run twice moves by more than that. A loop that accepts edits based on single runs will keep accepting noise and calling it progress.

The design

Each of these came from one of the papers or from that noise measurement.

1. Hold out whole repositories. The benchmark is SWE-bench Verified, scored on localization: did the agent name the file the real fix changed? I split it by repository, not by case. All xarray and sphinx cases (66) are held out, and the optimizer never sees those repos. If the optimizer learns something like “in sphinx, look in the autodoc module,” it gets no credit on the test set.

2. Measure the noise band in every run. Round 0 of every run evaluates the unchanged harness with at least two trials per case. Trial 1 vs trial 2 on each case shows how much the score moves when nothing changed. I bootstrap the standard deviation of that difference, and the noise band δ is twice that, scaled for how many trials each candidate gets. If a candidate’s gain is smaller than δ, it hasn’t shown anything.

3. Put the acceptance rules in code. The model doesn’t decide what gets accepted. These rules are adapted from RRSI’s coding setup:

floor       S' >= S* - δ                  never accept something clearly worse
clear gain  ΔS > δ   → accept if ΔC <= 0.10 + 44.5·ΔS   (a real gain may cost more)
in band     ΔS <= δ  → accept only if it is cheaper    (a gain inside noise counts for nothing)
guards      veto if the agent gives up more often, or breaks an always-pass case

The in-band rule matters most. RRSI’s coding setting gives zero weight to score changes inside the noise band, so inside the band the only way to win is to be cheaper. That makes lucky runs very hard to accept.

4. Compare case by case. The candidate and the current best harness run on the same cases, and every decision records which cases got better and which got worse. A +3 made of 8 cases up and 6 down is just noise.

5. Check for leakage before spending money. Every proposed edit goes through two checks:

• A regex denylist built from the benchmark: if any added line mentions a case ID, a repository name, or a file a real fix touched, it’s rejected.
• A second LLM reviews the edit against RRSI’s six reject classes: task leakage, no-op changes, gaming the grader, bundling several changes, memory leakage, and unbounded work. I added two more later for reasons that come up in Part 6.

The main test the critic applies: would this edit make sense on a JavaScript production incident from CloudWatch logs? That’s the agent’s real job. SWE-bench is just where it practices.

6. One edit per round, with memory. Each round the proposer reads the traces, where the money went, and the history of every edit so far and why it was accepted or rejected. It proposes one change, so every result can be traced to one change.

7. Get the rerun baseline for free. AI2’s test is that the evolved harness run once should beat the original harness run twice. Running a separate rerun arm costs a lot. But if the final held-out evaluation runs each harness three times per case, you can estimate the original’s best-of-2 score from those same three trials, without bias, using the pass@k estimator from the Codex paper. So the baseline costs nothing extra.

The loop

One round looks like this:

propose   an LLM reads failing traces, cost-by-source and the edit history → one edit + hypothesis
screen    structural validation, then the leakage critic (rejections go back for repair)
evaluate  the edited harness on every evolve case, 2 trials each  (the only expensive step)
decide    the rules above, in code, case by case against the current best harness
remember  hypothesis, diff, ΔS, ΔC and the verdict go into the history the next round reads

After the last round, a final phase runs the original and the final harness on the 66 held-out cases and writes a report: single-run localization, best-of-2, reliability (pass^k), and confidence intervals for each comparison.

The optimizer also had to run for hours without me watching, so it has checkpoints after every phase, a per-case result cache so a crash never pays for the same work twice, a hard budget cap in code, and tripwires that pause the run if its own measurements start to look broken. That gets its own post later in the series.

What I expected vs what happened

I expected the loop to find a few prompt improvements and the rules to keep them honest. That isn’t quite what happened:

• The first full run rejected every edit it proposed, and every rejection was correct.
• Before that, its first runs were spent working around a tool that was silently returning wrong answers.
• The biggest improvement early on came from me reading the optimizer’s traces, not from anything it proposed.
• The harness ran an open-weight model at a fifth of the cost, with better held-out localization than the frontier model it was built for.
• Once I worked out why the optimizer kept failing and fixed the method, it found changes that held up on repos it had never seen, and those changes now run in production.

The main thing I learned: in a self-improving system, most of the work is the evaluation design. The proposer is the easy part.

Next: Part 2: Before the Optimizer: The Tool That Lied, and Other Silent Failures.