Power Analysis, and the Run That Worked

Part 6 of the Self-Improving Harness series. Part 3 ended with every edit rejected, Part 4 with me finding retry instead of the optimizer, and Part 5 with the harness running DeepSeek-V4.1-Flash at a fifth of the cost. This post is about why the optimizer kept failing, and the run where it didn’t.

---

Another run that wasn’t working

After Part 5, I started the optimizer on DeepSeek. The idea was that a cheaper model that fails more often would leave more room for gains. Round 1 came back worse than the baseline, again with “submit sooner”, the sixth time that idea had come up. I stopped the run there and asked a different question: instead of trying another model or another idea, why was nothing ever getting through?

Measuring how much signal I had

I had a change I knew was real: retry, worth +8.3 points in the live run in Part 4. So I used it as a known answer. If the setup couldn’t reliably see retry, it couldn’t see anything smaller.

Using trials I had already paid for, I compared retry’s gain on the 51-case evolve set against the noise band for a 2-trial evaluation:

	Retry gain	Noise band	Gain ÷ noise
Sonnet, 51 evolve cases	+0.072	0.077	0.93
DeepSeek, 51 evolve cases	+0.075	0.079	0.95
Sonnet, hard cases only (34)	+0.069	0.100	0.69

This was the answer. Even retry, an 8-point change, sat right at the edge of the noise band on 51 cases, on both models. Realistic harness improvements are 5 to 10 points. So at this sample size, the optimizer could propose a good idea and the rules would still correctly say “not proven”. That’s exactly what happened to “submit sooner” in Part 3: +0.029, three times.

It wasn’t the model and it wasn’t the ideas. It was the sample size.

Why “just use the hard cases” doesn’t work

My first thought was to evolve only on the cases that fail, since that’s where there’s room to improve. I checked it with the data instead of arguing about it, and it made things worse (0.69 in the table above).

The reason is that most cases don’t carry any information. On Sonnet, 29 of the 51 cases passed all 3 trials and 10 failed all 3. Those cases can’t move, so they add no signal and no noise. Both the gain and the noise come from the in-between cases that sometimes pass. Filtering towards them raises the gain and the noise by the same amount. And Sonnet already passed most of the “hard” tier, so filtering just shrank the sample.

What actually helps is more of those in-between cases, and more trials: signal-to-noise grows roughly with the square root of (informative cases × trials). On Sonnet, only about 22 of the 51 cases were informative. That was the real limit.

Finding enough informative cases

This is where DeepSeek’s price mattered. At about $0.03 per diagnosis, running all 500 SWE-bench Verified cases twice was affordable ($39.09, covered in Part 5). That run also told me which cases were informative for DeepSeek specifically: 138 of the 434 cases outside the held-out repos had at least one failed first attempt.

So the next run, r9, would evolve on those 138 cases plus 6 easy cases as guards, instead of 51 cases that were mostly always-pass or always-fail.

There was a trap in this, though. I picked those 138 cases because they failed in that run. Some of them were just unlucky, and they’d score higher on any rerun, which is regression to the mean. If I had compared candidates against those same trials, every edit would have looked like an improvement. So r9 had to measure its own baseline on fresh trials.

It did, and the effect was large. The 138 cases scored 0.31 on the trials that selected them, and 0.49 on fresh trials. Comparing against the old trials would have given any edit a fake gain of about 18 points. With the fresh baseline, the noise band came out at 0.063, the tightest of any run, and a simulated retry gain was about 1.9 times the band. So a real change should now clear it.

Fixing the optimizer too

The sample size was the main problem, but the earlier runs had also shown specific weaknesses in the optimizer itself. Before r9 I fixed each one:

Change	The failure it answered
Settings shown as behaviour: value, allowed range and what it does	In Part 4, the optimizer never touched settings, even the one aimed at the exact failure it described
The proposer told the real acceptance rules for this run	Its prompt said gains were “hard to measure at this sample size”, which pushed it towards cost-only edits and wasn’t true anymore
A list of rejected ideas, and a near-duplicate check in the critic	”Submit sooner” had been evaluated six times, about $14 each on Sonnet
Two candidates per round	A critic rejection used to waste the whole round
A check for false claims about tools	In Part 3, an edit with a false claim about verify_symbol passed the critic
The leakage denylist covering all 500 cases	It only covered the original split’s cases

The run

r9 ran on DeepSeek with a $125 cap: the 138 informative cases plus 6 guards, 2 trials each, up to 6 rounds, starting with retry off, and ending with the held-out test on the 66 cases from repos the optimizer never saw.

Round	What the optimizer proposed	Change in pass rate	Change in cost	Result
1	Describe blast_radius as call sites the agent has already seen	+0.052	−7.8%	rejected, just inside both noise bands
2	Make submitting “the start of a correction loop”	+0.066	−4.8%	accepted
3	Turn on retry_on_no_submission	+0.080	+34.7%	accepted
4	Mention a common rejection reason at submission time	+0.003	+4.3%	rejected, no effect
5	(none: the proposer returned empty output)			invalid
6	Double the file read limit, 12,000 → 24,000 characters	+0.007	+0.3%	rejected, no effect

Round 2: “submit sooner”, finally measurable

Round 2 was the same idea that had been rejected six times before, now worded as a correction loop: the validator tells you exactly which field is wrong, the run ends when turns run out, so submit as soon as you have a file and a quoted snippet and use the remaining turns to fix it.

It scored +0.066 against a noise band of 0.063, and cost 4.8% less. The first accuracy-based acceptance in the whole project. The idea was never wrong. On 51 cases it showed up as +0.029 and couldn’t be proven. On 138 informative cases the band was tight enough to see it. That was the power analysis’s prediction coming true.

It was a very close call, though (+0.066 against 0.063), which is why the held-out test was the one that mattered.

Round 3: the optimizer proposed retry itself

This is the part I was hoping for. In round 1 and round 2, retry was the optimizer’s second choice. In round 3 it was its first choice. It wasn’t forced. The exploration rule, which pushes the optimizer onto settings after two rejections, had been reset by round 2’s acceptance, so it could have picked anything.

Its reasoning was the same one I used in Part 4: “88 of 288 runs (31.2% escalation) end with no accepted submit_diagnosis at all… the failure is stochastic rather than deterministic (many cases are PASS on one trial and FAIL on the other).”

It scored +0.080 on top of round 2’s change, with give-ups dropping from 30% to 16%. It cost 34.7% more, but a clear gain that size is allowed to cost more under the rules.

I want to be precise about this. The optimizer found retry after I changed how it sees settings and what it’s told about the rules. In the earlier run without those changes, it never touched settings. So the honest claim is that the improved optimizer found it, and the difference between those two runs shows the method change worked.

Round 5: my mistake

Round 5 was lost. The proposer returned an empty answer three times. My best guess is that it ran out of output tokens. It runs on Opus with an 8,192-token cap and thinking on, and my change to ask for two candidates per round roughly doubled how much it had to write. I didn’t log the stop reason, so I can’t confirm it. It’s on my list to fix.

A provider outage

During round 6, Together went down with a 503. The run paused itself cleanly: every finished case was cached and nothing was scored as a failure. The first session had run for 5.21 hours straight. I made the gateway ride out longer outages, and it resumed from the cache. In total r9 ran for 7.4 hours, did 2,129 agent runs and cost $102.10. Part 7 covers how the long-running side of this works.

The held-out test

On the evolve set, the two accepted changes took the pass rate from 0.510 to 0.656. But that’s on cases the optimizer was looking at. The real question was whether the changes would hold on 66 cases from repos it never saw, 3 trials each:

66 held-out cases	Original	Evolved
Single run	78.8%	89.9%
Passed all 3 trials	60.6%	78.8%
Gave up with no answer	19.7%	5.6%
Cost per diagnosis	$0.033	$0.040

+11.1 points, 95% CI +6.1 to +16.2. 20 cases improved, 2 got worse, 44 didn’t change. (The original scores 78.8% here and 81.1% in Part 5 on the same cases and model. Those are separate runs with different trials, 3 here and 2 there, and the gap is within noise.) The changes the optimizer proposed, and its own rules accepted, held up on unseen repos.

Then the test from AI2’s paper, the one that matters most: does it beat just running the original twice?

	Localization	Cost per diagnosis
Original, best of 2 runs	89.9%	$0.066
Evolved, single run	89.9%	$0.040

It’s a tie (difference 0.000, CI −0.04 to +0.04). The evolved harness doesn’t beat running twice on accuracy. It matches it at about 60% of the cost, and without needing a way to pick between two answers. That’s what I’d expect, since one of the two accepted changes is basically a smarter rerun. I don’t say it beats reruns, because it doesn’t.

Against Sonnet 5 with retry on the same 66 cases (79.5%), DeepSeek with the evolved harness is 10.4 points ahead (95% CI +3.8 to +18.2) at 4.7× lower cost.

One more check before shipping

Production diagnosis ran on Sonnet, and r9 evolved the harness on DeepSeek. So before promoting it, I ran the evolved harness on Sonnet on the same 66 held-out cases, 2 trials each.

It didn’t transfer. The evolved harness scored 77.3% on Sonnet against 79.5% for the original with retry, a difference of −2.3 points (CI −7.6 to +3.0), within the noise. So the prompt change helps DeepSeek and does nothing measurable for Sonnet. Retry does transfer: it was worth +8.3 points on Sonnet in Part 4.

So I shipped the evolved harness together with the model it was evolved on. Production diagnosis now runs DeepSeek-V4.1-Flash with the evolved harness. The lesson I took: changes found by evolution can be specific to one model, so re-check on every model you ship.

Why this run worked when the others didn’t

Looking back, three things made the difference:

1. Enough informative cases. 138 cases the model actually fails sometimes, instead of about 22. This was the big one.
2. A fresh baseline. Without it, regression to the mean would have handed every edit a fake 18-point gain.
3. A better proposer. Seeing settings as behaviour and knowing the real rules is what let it find retry on its own.

None of those are about the model being smarter or the ideas being better. “Submit sooner” was a real effect from the very first run. I just couldn’t measure it until I had enough of the right cases.

Next: Part 7: Running an Agent for 7 Hours: Tripwires, Watchdogs and a Provider Outage.