Fine-Tuning a Distillation Model on a Golden Dataset I Had to Build First

The dataset problem, before the fine-tuning problem

TriageAgent classifies a production error as real/noise/duplicate and assigns a severity (P0-P3), using a prompted Haiku call plus two tools (occurrence count, duplicate-PR check). It’s cheap and it’s accurate. The question I actually wanted to answer wasn’t “is it accurate” — it was “could a small fine-tuned model reproduce this judgment, and what would that actually cost.”

The obvious first move — fine-tune against the existing golden dataset — died in about five minutes of looking at the data. golden_dataset.jsonl had 512 rows and exactly 14-16 unique underlying scenarios repeated hundreds of times. Fine-tuning on that teaches a model to recognize 15 specific error strings, not to reproduce a severity judgment. I needed a dataset that actually spanned the decision space, and I wasn’t willing to hand-write the labels — a hand-labeled dataset would just be testing whether the model can imitate my guess at what the agent would say, not the agent’s real behavior.

Synthetic inputs, real judgments

The methodology that survived scrutiny: construct plausible synthetic input scenarios, grounded in a real error taxonomy (the kinds of error_type/service/occurrence-count combinations that actually show up in production), mock the two tools to controllable values, and then run the real TriageAgent — a real Haiku call, no shortcuts — against each one. Keep the case only if the actual output matches the category the scenario was built to target. If the model disagrees with my intended label, the model’s answer wins and the case gets discarded. Labels are never hand-fabricated; they’re always what the real agent actually said.

This is slower than writing labels by hand, and it should be — the whole point is that a “golden” dataset that doesn’t survive contact with the real model isn’t golden, it’s just my opinion with extra steps.

It also meant iterating on templates like a real experiment, not a data-entry task. The P0 templates got the worst initial match rate — 12.5% (1-2 of 12). Occurrence counts like 45-60 read to the model as “recurring but not catastrophic,” not “full outage,” even though my intent was P0. Rewriting with blunter impact language (“100% of requests are failing,” “no functionality works”) and lower, more acute-sounding counts (5, 8, 3, 4) brought that to 83%. P3 and noise templates had the opposite failure mode — the model weighed error content over the low occurrence count (“SyntaxError” sounds like a real bug regardless of frequency); explicit “isolated, one-off” and “expected, not a bug” framing fixed most of it. One template — a deprecation warning I intended as P3 — never budged, landing P2 in every single retry regardless of how I reframed it. At that point I stopped fighting the model’s judgment and did the honest thing: reclassified the template’s target category to P2 instead of writing increasingly contorted prompts to force a P3 out of it. If the model’s read is reasonable, the dataset should reflect that, not my original guess.

The real incidents that didn’t say what I expected

I’d also pulled real crash signatures out of the platform’s own CloudWatch history — a heap-exhaustion problem recurring since December 2024, a broken-deploy MODULE_NOT_FOUND episode, a downstream ECONNREFUSED outage, a SIGKILL. I assumed these would anchor the P0/P1 end of the dataset — they’re real incidents, they felt severe. Feeding their real historical peak occurrence counts into the agent (16, 15, 8, and 1 occurrences/24h — all well under the >100/24h P1 threshold) got P2/P2/P2/P3 back, every time.

I considered reframing the incident descriptions to push them toward P0/P1 — they’re “real” incidents, surely they deserve it — and talked myself out of it. That would be the exact same mistake as hand-fabricating a label: substituting my read of the incident’s importance for the agent’s actual judgment given the actual numbers. The frequency-based severity guide doesn’t care how dramatic an incident sounds; it cares how often it’s firing. Kept all four cases exactly as the agent scored them.

536 cases and one stratified split

Final dataset: 532 synthetic cases (P0: 53, P1: 72, P2: 132, P3: 93, noise: 84, duplicate: 98) plus the 4 real CloudWatch cases, split 80/20 by category with a fixed seed — 428 train, 108 held out, zero overlap. Converted to OpenAI’s chat-JSONL fine-tuning format, which turned out to be the right call for a reason I didn’t anticipate: it’s the same format Together AI and Fireworks both use.

The pricing detail that rewrote the cost story

Together AI’s LoRA fine-tune of Qwen/Qwen3.5-9B — 428 training cases, 3 epochs — cost exactly what it estimated beforehand: $4.00, training loss 0.597 → 0.551. Clean, cheap, fast (about 7 minutes end to end).

Then I tried to actually call the fine-tuned model and got a 400: “Unable to access non-serverless model… create a dedicated endpoint.” I’d assumed a LoRA fine-tune of a model with cheap serverless pricing ($0.17/$0.25 per 1M tokens) would inherit that pricing. It doesn’t. I checked whether this was a Together-specific limitation before believing it — Fireworks’ own documentation says the identical thing, in almost the exact same words: “Trained LoRA models… can only be deployed to on-demand (dedicated) deployments. Serverless deployment is not supported.” Not a platform quirk; this is how LoRA fine-tune serving works right now, industry-wide, outside of the managed (and, as of this project, largely closed-off) fine-tuning services.

That changes the entire framing of “is this cheaper than Haiku.” It’s not a per-token comparison anymore — it’s a dedicated GPU sitting there billed by the minute (Together: 1×H100 via a v2 dedicated-endpoint API; Fireworks: $8/hour for the same class of GPU as of this writing) versus a metered API call that costs nothing when idle. That’s a real, useful, honestly-different tradeoff to reason about — not a downgrade of the result, just the correct version of the result.

Running the actual eval

Spinning up a dedicated endpoint for evaluation only, on the clock, changes how you write the eval script: everything has to run in one shot, get torn down immediately whether it succeeded or not, and write results to disk incrementally in case something crashes mid-run. One more gotcha before the numbers were clean: Qwen3.5-9B thinks by default — burns tokens on hidden chain-of-thought before answering — and with a modest max_tokens that means the answer never arrives (finish_reason: length, content: None). Disabling it needed an explicit chat_template_kwargs: {"enable_thinking": false} on every call.

Against the 108 held-out cases — never seen during training — the fine-tuned model matched the real agent’s decision (real/noise/duplicate) 100% of the time, and severity (P0-P3) 95.4% of the time, at a 0.55-second mean latency. The five misses were all adjacent-severity boundary calls — a P2 read as P3, a duplicate/P1 read as duplicate/P2 — never a decision-category error. For a 428-case training set, that’s a genuinely good result, and an honest one: it says the categorical judgment generalizes cleanly, and the severity boundary is where a small model with this little data still has room to be sharpened.