The Best Idea Came From the Optimizer's Data, Not the Optimizer

Part 4 of the Self-Improving Harness series. Part 3 ended with the first full optimizer run rejecting all three of its edits. This post is about what I found when I went back through the data that run produced.

---

Looking at how the agent fails

After the run in Part 3, I had 662 agent runs worth of traces and no accepted edit. Instead of starting another run, I wanted to understand how the agent actually fails.

The answer was the same in every round. Almost every failure had one shape: the agent used up its turns without ever getting an accepted submit_diagnosis, then escalated to a human. Wrong-file answers were rare. And these failures were mostly random. The same case would fail on one trial and pass on the next with nothing changed. About a quarter of the cases passed sometimes and failed other times.

The optimizer had seen this too. Its answer, in round 2 and in two earlier runs, was a paragraph telling the agent to submit sooner. It never got through the noise band.

But a failure that is random and announces itself is exactly what a retry fixes. If a run ends with no accepted diagnosis, there’s no answer to lose. So you just run it again. There’s also nothing to choose between afterwards, because only one of the two runs has an answer. That’s different from “run it twice and keep the better one”, which costs double every time and needs some way to decide which answer is better.

Checking it for free first

Before spending anything, I checked the idea against trials I had already paid for. The held-out phase in Part 3 ran every case 3 times, and those trials are independent. So for every case I could pair up its trials and ask: if the first one ended with no answer, would the second one have passed?

Policy	Held-out (66 cases)	Cost
Single run	72.2%	1×
Retry only when there’s no answer	80.3%	1.34×
Always run twice, keep the better one	80.8%	2×, plus a way to pick

The retry got almost all of the rerun gain at about a third of the extra cost. About 27% of runs needed the retry.

That’s a simulation, though, and I didn’t want to claim it until I’d run it for real.

Running it for real

The harness got a new setting, retry_on_no_submission: if a diagnosis ends with no accepted submission, run it again, up to that many times. I ran the 66 held-out cases twice each on Sonnet 5 with it turned on. One run gives both answers, because each trial records the first attempt and the retry separately.

	Live (66 cases, 132 trials)	Simulated
First attempt only	71.2%	72.2%
With retry	79.5%	80.3%
Gain	+8.3 points (95% CI +3.8 to +14.4)	+8.1 points
Runs that needed a retry	27%	27%
Runs ending with no answer	27% → 19%
Cost per diagnosis	$0.139 → $0.188 (1.35×)	1.34×

The live run took 54 minutes and cost $24.75. The simulation predicted it within 0.8 points on every line, which also told me the evaluation code (the paired trials, the cache and the scoring) was working the way I thought.

The agent isn’t smarter with retry. Its failures were mostly bad luck, and a retry turns some of that bad luck into a second chance. The number of cases that end with “no answer, send it to a human” dropped by about a third.

Checking it on real production incidents

SWE-bench is Python open-source code. My agent’s real job is diagnosing a JavaScript production app from CloudWatch logs. So before shipping this, I replayed the 6 real production incidents I have ground truth for, each at the commit before its fix was merged, with retry off and on.

	Retry off	Retry on
Found the right file	4 of 6	5 of 6

The retry fixed exactly the incident that had ended with no answer, and didn’t make any other incident pick a worse file. Six cases is tiny, but it was the safety check I needed before turning it on in production. Retry is now on by default.

So why didn’t the optimizer find it?

This is the part I found most interesting. The optimizer never proposed anything like this, and at first that made sense: in that run, retry wasn’t something it could change. Its edit surface was prompts, tool descriptions and settings, and retry is control flow.

So I tested it directly. I added the retry setting to settings.json, turned it off, gave it a one-line description (“re-run a diagnosis that ended with no accepted submission”), and started a new optimizer run with no history to see if it would use it. The question was not “can it invent retry”, just “if the setting is right there, does it use it?”

It didn’t. In 2 rounds, it edited text both times and never touched settings:

• Round 1 was “submit sooner” again, as a prompt paragraph, the fifth time that idea had been proposed across all my runs. Its own evidence said exactly what was wrong: 34 of 102 runs used all 15 turns and never submitted. The setting it was ignoring targets exactly that failure.
• Round 2 found something real that I had deliberately not built, to see if it would find it: the agent wasting turns on repeated narrow reads of the same file (lines 2340–2410, then 2360–2400, then 2280–2320, each one stopping mid-function). Its fix was a tool description telling the agent to ask for a bigger window. It gained +6.9 points, just inside the noise band of 8.1, and cost 5% more, so it was rejected.

My best guess at why: the proposer’s prompt and evidence were mostly prompt text and agent traces, the settings file looked like numbers to tune rather than behaviour, and a model asked to reflect on an agent’s reasoning tends to propose reasoning fixes. It finds real problems, but it expresses every fix as text.

That run cost $45.24 and also exposed a gap in my own rules. Round 1’s “submit sooner” was accepted, for being 1.8% cheaper with no real change in pass rate. The same idea had been rejected four times before for being more expensive. The measured cost difference for that one idea ranged from −1.8% to +4.8% across runs, so a small cost difference was really just noise, and the rules had no noise band for cost, only for pass rate. I added a cost noise band after that, calibrated the same way as δ.

What I took from this

A few things changed how I approached the rest of the project:

• The best change so far came from me reading the optimizer’s traces, not from the optimizer. I want to be clear about that. The loop’s data was what made the idea obvious, but I was the one who saw it.
• Harness engineering isn’t only prompts. The biggest gain came from control flow, which a prompt-only optimizer can’t reach. And even when control flow is in its reach, a reflective proposer anchors on text.
• A retry beat rerunning, on cost. Running twice every time gets 80.8% at 2×. Retrying only when there’s no answer gets 79.5% at 1.35×, with no need to pick between answers.

This also pointed at something to fix in the optimizer itself: the proposer needed to see settings as behaviour, and needed to see the actual acceptance rules it was being judged by. When I made those changes later, the optimizer did propose retry on its own. That’s in Part 6.

Next: Part 5: The Harness Is Portable: An Open Model at a Fifth of the Cost.