The Harness Is Portable: An Open Model at a Fifth of the Cost

Part 5 of the Self-Improving Harness series. Part 4 covered the retry change I found in the optimizer’s traces. This post is about running the same harness on a much cheaper open-weight model.

---

Why try another model

By this point I had two questions I couldn’t answer with Sonnet 5 alone.

The first was about the harness. Everything so far had been tuned and measured on Claude. If the harness only worked well because of Claude, then it wasn’t really a good harness, it was a good fit for one model. I wanted to know how much of the result belonged to the harness itself.

The second was about the optimizer. On Sonnet, no edit could get through the noise band. The baseline was already fairly strong and the gains were small. A cheaper model that fails more often leaves more room for a real improvement to show up. So if an open model worked at all, it would also be a better place to run the optimizer.

The scout

I started small: a few open-weight models that Together serves serverless, on 17 cases, one trial each, for about $3.

Model	Pass rate	Cost per diagnosis	Ended with no answer
DeepSeek-V4.1-Flash	59%	$0.026	29%
MiniMax-M3	41%	$0.017	59%
GLM-5.3-Flash	0%	$0.004	100%

For comparison, Sonnet scored 82% on those same 17 cases. So DeepSeek was clearly worse on this small sample, but at a fraction of the cost, and with a lot of runs ending with no answer, which is exactly what retry fixes.

GLM’s 0% was not a real result. Every run had zero turns. Together was rejecting the calls because of a credit limit, worded differently from the billing errors I already handled, so my code counted them as the model failing. It would have been easy to write GLM off as useless. I fixed the code to treat that error as a billing stop instead of a model failure, the same kind of silent failure as in Part 2.

The same harness on DeepSeek

Next I ran DeepSeek on the 51-case evolve set from Part 3, 3 trials per case, with the byte-identical harness and code Sonnet had been measured on. No prompt changes, no model-specific tweaks, and retry off so it was a fair comparison with Sonnet’s baseline.

51 evolve cases, same harness	Pass rate	Ended with no answer	Cost per run
DeepSeek-V4.1-Flash	73.2%	22%	$0.031
Sonnet 5	63.7%	33%	$0.144

That’s +9.5 points (95% CI +1.0 to +18.3) at 4.6× lower cost. Sonnet’s best baseline in an earlier run was 68.6%, and against that the lead is about 5 points and not significant. So the careful way to say it is: DeepSeek matched Sonnet or did better, at about a fifth of the cost.

I made a mistake on the way here that’s worth mentioning. My first comparison picked the wrong Sonnet number. It grabbed a candidate’s evaluation instead of the baseline, because it took “the first folder” instead of selecting by the harness’s content. I caught it before reporting anything. Now every comparison selects what it compares by content, not by position.

All 500 cases

51 cases is a small sample, so I ran DeepSeek on all 500 SWE-bench Verified cases, 2 trials each, with retry on and every attempt recorded, so one run gives both numbers. It cost $39.09.

All 500 cases	Localization	95% CI	Cost per diagnosis
First attempt only	78.4%	75.4–81.4	$0.031
With retry	87.0%	84.3–89.5	$0.039

Retry was worth +8.6 points here too, about the same as on Sonnet (+8.3). It also made the agent more consistent: the share of cases it passed on both trials went from 69.0% to 82.2%.

The weakest repo was matplotlib (60.3%, and 70.6% with retry). The strongest was pytest (89.5% and 97.4%).

The important comparison is on the 66 held-out cases, from the two repos that no optimizer run had ever been tuned on:

66 held-out cases	First attempt	With retry	Cost per diagnosis (with retry)
DeepSeek-V4.1-Flash	81.1%	89.4%	$0.037
Sonnet 5	71.2%	79.5%	$0.188

With retry, DeepSeek is 9.8 points ahead (95% CI +2.3 to +18.2) at about a fifth of the cost per diagnosis. That lead is significant, and it points the same way as the evolve set.

”Isn’t it just memorized?”

This is the first thing I’d ask about any SWE-bench result. The issues and fixes are public, and a newer model may have seen them in training. A higher score could mean it remembers the answers.

SWE-bench can’t answer that, but my production incidents can. They come from a private codebase no model has trained on. I replayed the 6 real incidents I have ground truth for, each at the commit before its fix:

Found the right file	Retry off	Retry on
DeepSeek-V4.1-Flash	5 of 6	5 of 6
Sonnet 5	4 of 6	5 of 6

On private code, DeepSeek matched or beat Sonnet. With retry off, it even got one incident right that Sonnet missed in every run: a fix that touches two files, where Sonnet always picked the other file. Six cases is a small number, but it points the same way as the benchmark.

What the cost looks like

Price per token is not the same thing as price per diagnosis. What matters is what a whole diagnosis costs, and that depends on how many turns the agent takes and how much of each prompt is cached.

On DeepSeek, cached input costs $0.006 per million tokens, so resending the whole conversation every turn is nearly free. That flips where the money goes: model output is about 78% of spend on DeepSeek, compared with 53–57% on Sonnet. So on DeepSeek, turns and output are basically the only cost levers. Prompt length hardly matters.

Speed was about the same, roughly 2.5 minutes per diagnosis versus about 3 for Sonnet, estimated from throughput.

What changed because of this

A harness I built for Claude ran an open-weight model with no changes, and on repos neither model had been tuned on it was more accurate at a fifth of the cost. That told me the harness was doing real work, not just suiting one model.

It also gave the optimizer a better place to run. DeepSeek’s first attempts fail often enough, and cheaply enough, that a real improvement has room to show up above the noise. The next optimizer run, r9, ran on DeepSeek. That’s Part 6, where the optimizer finally produced changes that held up on unseen repos.

And it’s now what production runs. Diagnosis switched from Sonnet 5 to DeepSeek-V4.1-Flash, with the harness that came out of r9. Part 6 also covers one thing I found before shipping it: not everything the optimizer found carries over from one model to another.

Next: Part 6: Power Analysis, and the Run That Worked.