Before the Optimizer: The Tool That Lied, and Other Silent Failures

Part 2 of the Self-Improving Harness series. Part 1 covered the design: noise bands, held-out repos and a rerun baseline, all meant to stop the optimizer from fooling me. This post is about something I hadn’t planned for. Before the optimizer had a chance to fool me, my own tooling already was.

---

The optimizer’s first good idea was a symptom

In the optimizer’s first rounds, round 1’s proposal looked smart. The proposer read the failing trajectories and noticed a pattern:

Failing runs end with three to six consecutive verify_symbol_in_repo calls on spelling variants (def as_sql, Aggregate.as_sql, class Count(Aggregate)…) and never reach submit_diagnosis.

Its hypothesis was that the tool description was unclear about what input it accepted, so the agent kept retrying with different spellings. Its edit rewrote the description to explain the input format and say that one “FOUND” result was enough.

That’s a reasonable edit, backed by evidence. The rules rejected it because it raised the escalation rate past the guard, and I almost moved on. Instead I went and looked at what verify_symbol_in_repo had actually been returning.

87 of 87 calls had returned “NOT_FOUND”, and 85 of those symbols existed.

The cause was a one-character typo in the GitHub token the tool used. Every call got HTTP 401. The code-search wrapper caught the error and returned an empty list, and the tool reported the empty list as “this symbol does not exist in the repo.” The agent was following its instructions, which say never to name a symbol that doesn’t exist, so it tried another spelling. The loops the optimizer noticed were the agent reacting sensibly to a tool that was giving it wrong answers.

So the optimizer got the symptom right: the agent really was wasting turns in verify loops. But it proposed a prompt fix for an infrastructure bug. If that edit had been accepted, the harness would have learned to work around a broken tool, and the bug would have stayed hidden inside the instructions. Reading traces can find symptoms, but it can’t ask whether a tool is telling the truth. That needed me reading the raw tool outputs.

I’d seen this kind of mistake before in my own pipeline, where the fix agent kept adding a null guard at the crash site instead of fixing the function that produced the null. An optimizer that edits prompts is very good at producing symptom fixes, because the prompt is the only thing it’s allowed to change.

So the fix went in the tool, not the prompt:

• verify_symbol now searches the local checkout first, definitions before references and source files before tests.
• A failed lookup now returns VERIFY_ERROR instead of pretending the symbol doesn’t exist.
• The code-search call the tool depends on now raises an error on a non-200 response instead of returning an empty list.

After the fix, 0 of 120 lookups returned NOT_FOUND. On the same 17 cases, the pass rate went from 0.824 to 0.853 and cost per trial dropped from $0.121 to $0.109. So the broken tool had been costing both accuracy and money. I threw away every optimizer result recorded before the fix.

Three more tools with quiet failures

Once I started checking tool outputs instead of trusting them, I found more problems.

The call graph was from the wrong codebase. find_callers answers from a pre-built call graph. In production that’s the graph of my Node.js app. But in benchmark replays against Python repositories, it was still using the Node.js graph, so every find_callers call in every replay returned callers from the wrong project. Nothing crashed. The fix was to build the call graph from the checkout actually being diagnosed.

The retrieval tool never had an index. The agent had a search_codebase tool backed by vector search, and my own notes said diagnosis used hybrid retrieval. Actually, the production agent had been created without an index since week two of the project. Every call returned “RAG not configured,” and sometimes the agent called it three times in one run. The honest fix was to only offer the tool when an index exists, and to stop claiming the agent used it.

The budget couldn’t see its own spending. The optimizer has a hard budget cap in code. When I switched the evaluated model, the new model wasn’t in the price table, so as far as the budget knew, every call cost $0. A runaway loop could have spent without limit while reporting nothing spent. Now an unpriced model stops the run.

All of these kept running. A crash would have been found in minutes. A tool that returns wrong answers that look plausible, or a meter that reads zero, quietly ends up in every measurement after it. That’s why I think infrastructure that keeps running in a broken state is worse than infrastructure that crashes.

Where the money actually goes

The last piece of groundwork was cost tracking. The optimizer’s rules include cost, so I needed to know what a diagnosis costs and why. I captured every request in every trajectory, split each prompt by source (system instructions, task prompt, tool descriptions, each tool’s output, conversation history), and priced each part by billing type: cache reads, cache writes, uncached input and output.

The result surprised me. I’d read a well-known write-up on agent token efficiency where the biggest win was trimming the system prompt, and I expected the same here. Instead, the model’s own output was the biggest cost, 53–57% of spend. The task prompt was 16–20% and file contents were about 10%.

That changed two things about the optimizer.

First, prompt caching was the biggest free win. The agent resends its whole prefix every turn, and adding cache markers took the cache hit rate from 0% to 92–95%. Cost per trial also fell from $1.64 to $0.12 over the same period, but that mixes caching with a model upgrade, so I don’t count it as a caching result.

Second, once the prefix is cached, output is what matters. The proposer is told where the money goes, and told never to ask the agent to “be brief,” because that just makes agents reluctant to do the work. Instead it’s pointed at what the harness asks the model to write, like fields repeated on every resubmission and turns spent re-reading files.

What I’d check before trusting an optimizer’s numbers

If I did this again, I’d do these before the first optimizer round:

1. Sample raw tool outputs and look for answers that are suspiciously uniform. 87 of 87 of anything is a bug, not a finding.
2. Make every failure loud. An empty result from a failed call should be an error, never an answer.
3. Check that everything the agent reads belongs to the task being run: the index, the graph, the checkout.
4. Stop the run when a price is unknown instead of counting it as zero.
5. Measure cost by source before trying to reduce cost, since where the money goes probably isn’t where you’d guess.

None of this is exciting, but every result in the rest of the series depends on it. Once the tooling was fixed, the first full optimizer run rejected every edit it proposed. That’s Part 3.

Next: Part 3: Noise Is the Enemy: Why Every Edit Got Rejected.