Engineering blog
Notes from building Remediate Labs — agent architecture, retrieval design, debugging stories, and the unglamorous decisions that make production agents actually work.
Series
All posts
Designing a Self-Improving Agent Harness: How to Tell Real Gains from Noise
I wanted an agent that improves my diagnosis agent's harness. The research said it would mostly learn to overfit, so most of the design is about catching that.
Noise Is the Enemy: Why Every Edit Got Rejected
The first full optimizer run proposed three edits and rejected all three. Every rejection was correct, and the reasons were more useful than an accepted edit would have been.
Power Analysis, and the Run That Worked
Every optimizer run had failed for the same reason, and it wasn't the model or the ideas. Once I fixed how many cases it measured on, it found two changes that raised held-out localization from 78.8% to 89.9%.
Running an Agent for 7 Hours: Tripwires, Watchdogs and a Provider Outage
A long unattended run is also a long time for a bug to burn money. This is the machinery that let the optimizer run for hours, pause itself when something broke, and pick up where it left off.
The Best Idea Came From the Optimizer's Data, Not the Optimizer
Three rounds of prompt edits couldn't beat the noise. The traces those rounds left behind pointed at a small control-flow change worth +8 points, and the optimizer still didn't find it when I gave it the setting.
The Harness Is Portable: An Open Model at a Fifth of the Cost
I built the harness for Claude. I ran it unchanged on DeepSeek-V4.1-Flash, and it was more accurate than Sonnet 5 on repos neither had been tuned on, for about a fifth of the cost per diagnosis.
Before the Optimizer: The Tool That Lied, and Other Silent Failures
The optimizer's first good idea turned out to be a workaround for a bug in my own tooling. Before any of its results meant anything, I had to fix what it was measuring.
Testing and Observability for Code RAG
Two ways to know your RAG is working: a recall harness you run before you ship, and chunk-level tracing that shows what actually retrieved in production. Neither replaces the other — and this project already had half of each, in a shape worth being precise about.
Cross-Encoder Re-Ranking — A Real Negative Result
This technique didn't exist anywhere in the codebase before this post. Built it for real, measured it for real, and it makes the hardest query worse, not better. Here's exactly why, and what a two-stage retrieval architecture can't recover from.
Document Registry — Keeping the Index Honest
Upsert is insert-or-update by ID. It doesn't delete old chunks when a function is refactored. Without a registry, stale vectors accumulate silently until your LLM is reading code that was deleted six months ago.
Fixing the Chunking Split with Function-Boundary Chunks
The function name query score jumps from 0.49 to 0.71. A function that was buried at rank 9 moves to rank 3. Here's what changes when each function gets its own chunk — and why the extractor ended up being a bigger decision than expected.
Hybrid Search — Rescuing the Coin Flip, Not Fixing It
For the run where the description happened to name the error, hybrid search pushes the target from rank 8 to rank 2. For the run where it didn't, hybrid search does nothing at all — there's no token to boost. Both are real, measured outcomes of the same technique.
When to Re-chunk, Re-index, and Re-embed
Three operations that sound similar. They are not. Each has different triggers, different costs, and different consequences if you get them wrong — including one mistake this exact codebase's own index actually made.
Search Quality — Two Failure Modes and Why They're Different
One failure is a chunking problem. The other is a vocabulary gap that better chunking can't fix. Here's what the scores reveal about code RAG's limits.
Closing the Vocabulary Gap with LLM-Generated Descriptions
The error query went from not found to visible — sometimes. Sampling the same real model call eight times on the same function shows the fix works about half the time, not reliably. Here's the honest version of that story.
What Actually Gets Indexed
A 1080-line file produces 27 chunks. Most boundaries fall mid-function, with no awareness of code structure. Here's exactly what gets lost — and why it matters for search quality.
Fine-Tuning a Distillation Model on a Golden Dataset I Had to Build First
A skewed dataset, a synthetic-inputs-real-judgments methodology, and the pricing detail that rewrote the whole cost story.
We Built a Call Graph Because Our Agent Kept Breaking Callers It Never Knew About
The agent found one caller via GitHub search, patched it, and shipped. Two other callers broke in production. The fix was correct. The picture was incomplete.
Five Data Structures for a Call Graph — Why We Chose Hash Map of Sets
The adjacency matrix sounds clever. It's wrong for sparse graphs. The edge list is right for serialization and wrong for queries. Here's what each actually costs.
Token Cost Engineering in Agent Loops — Prompt Caching and State Pruning
Output tokens cost 5× more than input. In agent loops, both compound. Two targeted fixes — prompt caching and state pruning — cut real, measured cost on the diagnosis step by ~12%, with a separate, real dollar saving from state pruning on the fix-generation step.