Running an Agent for 7 Hours: Tripwires, Watchdogs and a Provider Outage

Part 7, the last part of the Self-Improving Harness series. Part 6 covered r9, the run that produced the result. This post is about the part underneath all of it: what it takes to leave an agent running for hours and trust what it did.

---

The worry

The optimizer’s longest run, r9, went for 7.4 hours in total, with its first session running 5.2 hours straight before a provider outage paused it. Before any of that, I had a simple worry: if something breaks at hour one, I don’t want to find out at hour eight with the budget gone and the results useless.

That had already nearly happened. In Part 2, a broken tool returned “not found” on every call, and two whole runs were scored on it before I read the traces. A long unattended run can spend its entire budget the same way.

So before the long runs, I built the machinery for this. Most of it came from something that went wrong first.

Never pay for the same work twice

The first piece is checkpoints. After every phase, the run writes its state to disk atomically, and resuming is the same command again. Every finished case is also cached, keyed by the harness’s content and the number of trials. So a crash only costs the cases that were in flight.

There’s one gotcha I hit several times. The cache is keyed on the harness, not on the agent’s code. After I fixed a bug in DiagnosisAgent, the old cached results were silently stale, because the harness hadn’t changed. The only safe thing was to start that run fresh. If I did it again, I’d put a hash of the relevant code into the cache key too.

A budget that can’t be fooled

The budget cap lives in code, not in my head. Each case reserves an estimate of its cost before it starts. When the budget is getting close, the parallel lanes wait for others to finish instead of stopping the whole run.

The first version didn’t wait. With 8 lanes each holding a reservation, the 8th lane found no room and stopped the entire run, at $0 spent.

The budget also fails closed. If a model has no price in the table, the run stops, because otherwise every call reads as $0 and the cap can’t see the spending. Provider problems like billing errors and outages stop the run too, but resumably, and nothing is scored as a model failure.

Tripwires

The worst failures in this project didn’t crash anything. They looked like data. So while cases finish, the run checks its recent results against what a healthy run looks like, every 10 finished cases over the last 20, and pauses if something is off:

• a tool failing far more often than normal (errors, “not found”, empty results)
• replays ending in errors or timeouts instead of a verdict
• every guard case failing, which means the easy path is broken
• cost per run far above normal, or exactly zero (a missing price)
• the baseline’s pass rate collapsing to under half of normal

“Normal” comes from measured runs, not guesses, and the thresholds are deliberately loose. A tripwire that fires on ordinary noise would end good runs for nothing. Every real breakage I had seen was all-or-nothing, like 100% “not found”, not a slow drift.

Before trusting them, I backtested them on old runs. They stayed silent on all the healthy data, and on the run with the broken tool they fired at the very first check: “verify_symbol_in_repo failing on 55/55 recent calls”. That bug had invalidated two whole runs. With tripwires it would have been caught after 10 cases, for about $2.

They have a limit, though, and I want to be honest about it. They would not have caught the other big bug from Part 2, the call graph built from the wrong codebase. That tool returned wrong answers, not failures, so there was no failure rate to trip on. Tripwires catch breakage that looks like failure. They don’t catch breakage that looks like data. Reading the first round’s traces by hand is still part of the process.

A smoke test before the long part

Before the expensive part of a run starts, a smoke stage replays 5 cases once and checks that every one produced tool calls, a verdict and a cost. In r5 it took 62 seconds and $0.40, and all 5 passed. Only then does the real run begin. It’s a cheap way to find out the setup is broken before spending an hour finding out.

The process that said it had stopped

This one surprised me. While checking on an earlier run, I found a process that had been hung for 18 hours. Its state file said it had stopped cleanly on its budget, with $0 spent. But the process itself had never exited. It was sitting at 0% CPU with no network connections.

It didn’t cost any money, but in a long unattended run, a hang like that silently wastes the whole window. “The state file says stopped” is not the same thing as “the process stopped”.

So every run now has a watchdog. It writes a heartbeat every minute. If there’s no progress for 20 minutes, it dumps the stack traces, saves state and exits, so the run can be resumed and I can see where it was stuck.

Making it fast enough

The 500-case run in Part 5 started at 20 cases in parallel, but finished only about 50 cases in its first 57 minutes. At that rate it would have taken 8 to 9 hours. The process was sitting at about 98% of one CPU core, so adding more parallel cases did nothing.

I measured where the time went instead of guessing. On a Django checkout of about 7,000 files, one diagnosis spent about 25 seconds in the grep tool, which read and scanned every file about 6 times per diagnosis, plus about 7 seconds building the call graph. That’s around 85 seconds of CPU per Django case, all on one core, and Django is 231 of the 500 cases.

The fix was to let git grep find candidate files first. It runs in C, off Python’s global lock. Then the unchanged Python matching runs only on those candidates. I also cached call graphs per commit, since the trials and retries of one case share a commit.

I didn’t want a faster tool that gives different answers, so I checked it by replaying 210 real recorded grep and lookup calls against their checkouts, old path against new. 0 mismatches, 5.9× faster.

A lock that only worked in one process

One replay in the 500-case run logged that it couldn’t prepare its git checkout, and then diagnosed anyway, without the code. That result would have counted as the model failing.

The cause was git itself. After many fetches into a shallow clone, git starts a background cleanup that holds a lock file, and a second process was fetching into the same clone. My lock only worked inside one process. I turned git’s automatic cleanup off for these clones, added a file lock that works across processes, and added retries on lock errors. Most importantly, a replay now prepares its checkout first and fails loudly if it can’t. It’s retried twice, then the run pauses. A replay without its checkout is never scored.

It’s the same lesson as the broken tool: infrastructure that fails but lets the agent keep running is worse than infrastructure that crashes, because it gets scored as the model being wrong.

The outage

In r9, all of this got tested for real. At 21:41 UTC, during round 6, Together returned a 503, “Service unavailable”. The gateway retried 3 times over about 7 seconds, which wasn’t enough. Then the classifier correctly recognised a provider outage, not a model failure, and the run paused itself cleanly: the current phase was kept, every finished case was already cached, and nothing was scored as a failure.

The first session had run 5.21 hours straight, 1,493 agent runs and 72 tripwire checks, all clean. It only ended because of the outage.

I changed the gateway to retry 6 times with backoff capped at 30 seconds, about 90 seconds in total, checked that Together was answering again, and resumed. The second session picked up from the cache, finished round 6 and ran the held-out test. In total: 7.4 hours, 2,129 agent runs, $102.10.

That run had also paused once at the very start, after 8 cents, because the credit I’d just added hadn’t registered on Together’s side yet. The same billing check caught it and paused cleanly. I confirmed the balance was live and restarted.

Being able to see it

The last piece was being able to check on a run without reading log files. The dashboard has an optimizer page that reads the run’s files and shows whether the run is alive (from the heartbeat), how much it has spent, the baseline and noise band, the tripwire results, the evaluation in progress against the current best harness on the same cases, and every round with its hypothesis, diff and reason.

What I’d do differently

Looking back across the whole series, a few things I’d do earlier next time:

• Run the power analysis before the first optimizer run, not after several of them. A known-real change measured against the noise band would have told me on day one that 51 cases couldn’t show a 5-point gain.
• Read raw tool outputs before trusting any result. Most of the worst problems in this project started with a tool that quietly returned wrong answers.
• Put the code’s hash in the result cache key, so a bug fix can’t silently leave stale results behind.
• Log the model’s stop reason on every call. The lost round in r9 was probably the proposer running out of output tokens, and I couldn’t confirm it because I wasn’t logging why it stopped.

The optimizer itself turned out to be the easy part. Most of the work was measuring honestly and keeping a long run from quietly going wrong.