Noise Is the Enemy: Why Every Edit Got Rejected

Part 3 of the Self-Improving Harness series. Part 1 covered the design and Part 2 covered the tooling bugs I had to fix before any result meant anything. This post is about the first full run of the optimizer, and why it didn’t accept anything.

---

The run

Once the tools were fixed, I wanted one clean run from start to finish with nobody touching it: measure the original harness, let the optimizer propose edits, then test the result on repos it had never seen. I gave it a $250 cap and up to 8 rounds.

The setup:

• Evolve set: 51 SWE-bench Verified cases the optimizer could see and learn from
• Round 0: the unchanged harness, 3 trials per case, to measure the baseline and the noise
• Each round: one proposed edit, evaluated with 2 trials per case against the current best harness
• Held-out set: 66 cases from two repos (xarray and sphinx) the optimizer never saw, 3 trials each

It ran for 3 hours 32 minutes in one session: 662 agent runs, $92.57, no restarts and no interventions. The rounds stopped after 3 because of a stall rule: three rejections in a row ends the rounds. Nothing was accepted.

How big is the noise?

Round 0 ran the same harness three times on every case. The average pass rate was 0.686, but 12 of the 51 cases flipped between pass and fail across those three trials, with nothing changed. About a quarter of the cases passed sometimes but not every time.

From those repeated trials the optimizer calculates its noise band, δ. For each case it compares trial 1 with trial 2, bootstraps how much the overall score moves when nothing changed, and takes twice that, scaled for the number of trials a candidate gets. With 2 trials per candidate on 51 cases, δ came out at 0.075. So an edit had to move the pass rate by more than 7.5 points before its gain counted as real.

For comparison, RRSI’s noise band was 0.017, measured from 178 trials. Mine was more than 4 times wider. That’s the honest cost of running this on a budget with 51 cases, and it turned out to decide everything that followed.

One check I was happy about: I had measured this same harness in an earlier pilot run, and the two independent baselines agreed within 0.01 (0.657 in the pilot and 0.667 here, using 2 trials each). So the measurement itself was stable. The noise was real, not a bug in how I was measuring.

The three rounds

Round	What it changed	Idea	Change in pass rate	Change in cost	Result
1	tool description	pass bare names to verify_symbol	−0.039	+2.1%	rejected
2	task prompt	submit sooner; a first attempt is cheap and fixable	+0.029	+4.8%	rejected
3	task prompt	shorten the prompt: turn past bug stories into rules	−0.049	+4.4%	rejected

All three were inside the noise band, and all three cost more than the harness they were trying to replace. Inside the band the only way to win is to be cheaper, so each one lost. Rounds 1–3 took 2 hours 28 minutes and about $44.50 together.

Each rejection taught me something different.

Round 1: half right, and half false

The optimizer noticed that stack traces give qualified names like Aggregate.as_sql, and proposed telling the agent to pass bare names like as_sql instead.

When I checked this against the code, half of it was right. My grounding gate checks the function name with a plain substring test, so Aggregate.as_sql gets rejected even when as_sql is in the file. That’s a real inconsistency between two of my own checks. But it only happened 3 times in 162 submissions, not “often” like the proposal said.

The other half was false. The edit claimed verify_symbol returns NOT_FOUND for qualified names. It doesn’t. It already reduces them to the bare name and says so in its output. So the edit would have written something untrue into a tool description, and my leakage critic passed it, because it checks for leakage and gaming, not whether a claim about a tool is true.

The acceptance rules threw it out anyway, but only because it wasn’t cheaper. A cheaper false description could have been accepted. I added a check for false claims about tools to the critic later (it comes up again in Part 6). The real fix for the inconsistency is one line in the gate, which the optimizer can’t touch because code is outside what it’s allowed to edit. It’s the same pattern as the tool bug in Part 2: the optimizer finds the symptom and proposes a prompt fix, and the actual cause is in code.

Round 2: +0.029, three times

Round 2 was the one I kept thinking about. The biggest failure mode in the whole project is the agent running out of turns without ever getting an accepted diagnosis. In round 0, 43 of 153 runs ended that way, and those were also the most expensive runs. So the optimizer proposed telling the agent that a first attempt is cheap and can be fixed, and to submit once it has a file, a quoted snippet and a verified symbol.

It scored +0.029, inside the band of 0.075, and cost 4.8% more. Rejected.

What made it interesting is that the same idea had come up before. In two earlier runs the optimizer had proposed it with different wording, and it scored exactly the same:

Run	Trials per case	Change in pass rate	Noise band	Result
First run (on the broken tool)	1	+0.029	0.117	rejected
Pilot	1	+0.029	0.117	rejected
This run	2	+0.029	0.075	rejected

Every one of those rejections is correct on its own, because each result is inside its own noise band. But getting the same small positive number three times in a row probably means there’s a real effect of about 3 points, and 51 cases just can’t see it.

That told me two things. First, the rules judge each experiment alone, and repeating the same result across experiments is information the rules don’t use. Second, the problem wasn’t the idea, it was that the evolve set was too small to measure an effect that size. I didn’t know how to fix that yet. Part 6 is about how I did.

The optimizer had also been shown that this idea was rejected before, and proposed it again anyway. The edit history is advice, not a ban. That’s reasonable, since re-testing ideas is fine, but each repeat cost about $14 to evaluate. I added a near-duplicate check to the critic later too.

Round 3: the cost cut that raised cost

Round 3 was the optimizer’s own idea for cutting cost. The task prompt is about 16% of spend and is resent every turn, and it has about 3,500 characters of stories about past production bugs. The edit replaced each story with the one rule it teaches.

It made things worse on both counts. The pass rate dropped 0.049, the agent gave up more often (28.1% to 33.3%) and took more turns, and cost went up 4.4%.

The reason is prompt caching. That 16% of spend is mostly cache reads, which cost a tenth of the normal input price. So cutting 3,500 characters saves very little per turn. But anything that makes the agent take extra turns costs full price, mostly in model output, which is the most expensive part. The well-known advice to trim your system prompt assumes you’re paying full price for it. With 92–95% of the prompt cached, the thing that matters is fewer turns and less output.

It also showed the stories weren’t dead weight. Removing them made the agent worse even on SWE-bench, which has none of the production code those stories are about.

What the held-out test said

After the rounds, the run tested the original harness on the 66 held-out cases, 3 trials each:

	Held-out
Single run	72.2%
Best of 2 runs	80.8%
Cost per run	$0.134

Running the agent twice and keeping the better answer was worth +8.6 points at twice the cost. Nothing the optimizer proposed came close to that. So on this agent, with Sonnet 5 and 51 cases, simple reruns beat reflective harness evolution. That’s the same thing AI2’s paper found, reproduced on a production agent.

The held-out score (72.2%) was also close to the evolve set’s (68.6%), so the baseline held up on repos it had never seen.

Why I count this as a good result

When the run finished, I didn’t have an improved harness. But I did have some useful things:

• The rules did what they were built to do. They rejected three edits that were inside the noise and more expensive, including one with a false claim about a tool that nobody had to catch by hand.
• The long-running setup worked. 3.5 hours, no restarts, every tripwire check clean, and it shut itself down properly.
• I knew where the room for improvement was. About a quarter of cases pass sometimes and fail other times, and the “submit sooner” result kept pointing at a real effect too small for 51 cases to measure.

The obvious move would have been to loosen the rules until something got accepted. That would have given me an accepted edit and a worse agent. I wanted to understand why nothing could get through, and whether the answer was a better idea or a better measurement. It turned out to be both: the better idea came from reading these traces myself (Part 4), and the better measurement came from working out how many cases the optimizer actually needed (Part 6).

Next: Part 4: The Best Idea Came From the Optimizer’s Data, Not the Optimizer.