#evals
8 posts tagged evals.
Designing a Self-Improving Agent Harness: How to Tell Real Gains from Noise
I wanted an agent that improves my diagnosis agent's harness. The research said it would mostly learn to overfit, so most of the design is about catching that.
Noise Is the Enemy: Why Every Edit Got Rejected
The first full optimizer run proposed three edits and rejected all three. Every rejection was correct, and the reasons were more useful than an accepted edit would have been.
Power Analysis, and the Run That Worked
Every optimizer run had failed for the same reason, and it wasn't the model or the ideas. Once I fixed how many cases it measured on, it found two changes that raised held-out localization from 78.8% to 89.9%.
Running an Agent for 7 Hours: Tripwires, Watchdogs and a Provider Outage
A long unattended run is also a long time for a bug to burn money. This is the machinery that let the optimizer run for hours, pause itself when something broke, and pick up where it left off.
The Best Idea Came From the Optimizer's Data, Not the Optimizer
Three rounds of prompt edits couldn't beat the noise. The traces those rounds left behind pointed at a small control-flow change worth +8 points, and the optimizer still didn't find it when I gave it the setting.
The Harness Is Portable: An Open Model at a Fifth of the Cost
I built the harness for Claude. I ran it unchanged on DeepSeek-V4.1-Flash, and it was more accurate than Sonnet 5 on repos neither had been tuned on, for about a fifth of the cost per diagnosis.
Before the Optimizer: The Tool That Lied, and Other Silent Failures
The optimizer's first good idea turned out to be a workaround for a bug in my own tooling. Before any of its results meant anything, I had to fix what it was measuring.
Fine-Tuning a Distillation Model on a Golden Dataset I Had to Build First
A skewed dataset, a synthetic-inputs-real-judgments methodology, and the pricing detail that rewrote the whole cost story.