Testing and Observability for Code RAG

Part 9 of the Code RAG series. Parts 1–8 built a pipeline that indexes correctly, retrieves relevantly, and rebuilds safely. This post covers verification: an offline recall harness you run before shipping any change, and production tracing that shows what’s actually happening at query time.

---

The two verification problems

Building a RAG pipeline well is not enough. You also need to know it’s working — and there are two distinct moments when it can fail silently:

Before you ship: a change to chunking, indexed text, embedding model, or retrieval parameters improved some queries and broke others. Without a test suite, you won’t know until a user reports a wrong answer.

After you ship: a query returns the wrong chunks. The LLM faithfully reasons over that wrong context and produces a confident but incorrect answer. Without chunk-level tracing, this looks identical to an LLM hallucination from the outside.

---

Offline: a recall@k eval harness — some of this already exists

The metric is recall@3: for a given query, is the correct chunk anywhere in the top 3 results? Not rank 1 — rank 1 is too strict for a system that passes all three results to an LLM. If the right chunk is in the top 3, the model has the information it needs.

Recall@3 = (queries where correct chunk is in top 3) / (total queries)

Binary per test case. No threshold guessing, no partial credit. Regressions are immediately visible.

This methodology already exists in this codebase — just not for code chunks. scripts/eval_rag.py runs almost exactly this pattern today, against the incident collection: for each of three real incident types, it fires an exact-wording query, a paraphrased query, and a root-cause-angle query, and checks whether the right incident lands in the top 3. The three-angle structure below isn’t a new idea for this project — it’s an existing, real methodology that’s never been pointed at code search specifically.

Three query angles per function

For each target function, write queries from three angles:

Angle 1 — Exact identifier. The function name verbatim. The easiest case: the token appears literally in the indexed chunk.

Angle 2 — Paraphrase. What the function does in plain language, without the name. Tests whether the embedding model captures intent, not just identifier presence.

Angle 3 — Root cause / error context. The language a diagnosis agent uses when it arrives with a theory — symptoms, no function name. From Part 2, this is the hardest case: code RAG has a vocabulary gap between what a function does and what it can fail with. Whether this angle passes at all is, per Parts 4–6, close to a coin flip for a function whose description didn’t happen to name its specific failure mode.

The test suite — against the real API, not an imagined one

import pytest
from app.services.rag import RAGService

# Canonical functions — real, currently indexed, stable reference points
MOVE_REMOVE_S3   = "moveAndRemoveFileFromS3"  # routes/services/image.js
CLASSIFY_FIELDS  = "classifyFields"           # constants/prankCheckerOpenAI.js


@pytest.fixture(scope="module")
def rag():
    return RAGService()


def top3_functions(results: list) -> set[str]:
    # RAGService.search() returns list[CodeChunk] — a dataclass, not a dict.
    # .get(...) / results[0]["score"] don't exist on it; use attribute access.
    return {r.function_name for r in results[:3] if r.function_name}


# --- moveAndRemoveFileFromS3 ---

@pytest.mark.asyncio
async def test_s3_exact_identifier(rag):
    results = await rag.search("moveAndRemoveFileFromS3", n_results=3)
    assert MOVE_REMOVE_S3 in top3_functions(results)

@pytest.mark.asyncio
async def test_s3_paraphrase(rag):
    results = await rag.search("copy S3 object to new key then delete the source", n_results=3)
    assert MOVE_REMOVE_S3 in top3_functions(results)

@pytest.mark.asyncio
async def test_s3_root_cause(rag):
    # Hardest angle — per Part 4, passes only if this function's generated
    # description happened to name its specific failure mode. Expect this
    # one to be flaky across re-indexes, not a stable pass/fail.
    results = await rag.search("S3 NoSuchKey missing key error", n_results=3)
    assert MOVE_REMOVE_S3 in top3_functions(results)


# --- classifyFields ---

@pytest.mark.asyncio
async def test_classify_exact_identifier(rag):
    results = await rag.search("classifyFields", n_results=3)
    assert CLASSIFY_FIELDS in top3_functions(results)

@pytest.mark.asyncio
async def test_classify_paraphrase(rag):
    results = await rag.search("classify moderation fields from an OpenAI response", n_results=3)
    assert CLASSIFY_FIELDS in top3_functions(results)


# --- Precision ---

@pytest.mark.asyncio
async def test_s3_query_does_not_return_classify(rag):
    results = await rag.search("S3 copyObject deleteObject bucket key", n_results=3)
    assert CLASSIFY_FIELDS not in top3_functions(results)

@pytest.mark.asyncio
async def test_classify_query_does_not_return_s3(rag):
    results = await rag.search("OpenAI moderation response field classification", n_results=3)
    assert MOVE_REMOVE_S3 not in top3_functions(results)


# --- Noise floor ---

@pytest.mark.asyncio
async def test_unrelated_query_low_score(rag):
    results = await rag.search("OAuth token refresh middleware", n_results=3, min_score=0.0)
    if results:
        assert results[0].score < 0.55, f"Noise floor too high: {results[0].score}"

Every attribute access above (r.function_name, r.score) matches CodeChunk’s real fields in app/services/rag.py. search() is genuinely async, so every test needs @pytest.mark.asyncio and await — a detail worth getting right, since a version of this suite that skipped it wouldn’t run at all.

A correction worth naming plainly: an earlier draft of this post used a class called CodeRAGService and treated search results as dicts (r.get("function_name", ""), results[0]["score"]). Neither exists. The real class is RAGService; the real return type is a dataclass accessed by attribute. Code written against the imagined API would fail immediately with AttributeError or TypeError the first time it ran — a good reminder that a test suite sketch that’s never actually executed against the real code carries exactly the same risk this whole reconstruction has been about: it reads as correct right up until someone runs it.

Also worth correcting: an earlier draft used runValidationCheck from a file called app/constants/validationOpenAI.js as a second canonical function. Neither the function nor the file exists anywhere in the target app. classifyFields is real, but it lives in constants/prankCheckerOpenAI.js — a moderation/content-checking file, not a validation one. The mixup traces to a real source, not nowhere: scripts/eval_rag.py’s own incident-eval query text uses “classifyFields” and “validationOpenAI” as descriptive vocabulary for an unrelated SyntaxError test case. It’s an easy mistake to make reading past code quickly — the words are real, they just don’t mean what they look like they mean out of context.

Baseline and regression use

Run this before and after every change to chunking, indexed text, or retrieval parameters. A drop from a passing suite to a failing one after a change tells you exactly which query angle broke. test_s3_root_cause is the most valuable single case in the suite — not because it’s expected to pass reliably (per Parts 4–6, it won’t, about half the time, through no fault of the code), but because tracking its pass rate over many re-indexes is a real, honest signal for whether the description-enrichment step is doing its job on average, in a way a single run never could be.

---

Online: production tracing — largely already shipped, via a different API than described

The eval harness tells you retrieval was working when you shipped. It can’t tell you what’s happening to a specific production query right now.

The invisible failure

Production RAG systems fail in ways that look like LLM problems but are actually retrieval problems. The answer is confidently wrong not because the model hallucinated — it faithfully reasoned over the wrong context. Without end-to-end tracing, you cannot distinguish “the model made something up” from “the model correctly summarized a chunk that shouldn’t have been retrieved.”

What this project actually has

An earlier draft of this post proposed adding OpenTelemetry spans and a Langfuse @observe decorator, framing this as new work needed because “RAG retrieval sits outside the agent loop, so it doesn’t get traced automatically.” That framing is backwards. This project doesn’t use OpenTelemetry at all — no opentelemetry package anywhere in requirements.txt, no import of it anywhere in the codebase. And retrieval tracing isn’t missing — it’s already implemented, using the Langfuse SDK’s own span API directly, not the @observe decorator style:

# Real, current code — app/services/rag.py
with lf.start_as_current_observation(
    name="rag.search",
    as_type="span",
    input={"query": query, "n_results": n_results, "min_score": min_score},
) as obs:
    chunks = await _run()
    obs.update(
        output={"num_results": len(chunks), "top_score": chunks[0].score if chunks else None},
        metadata={
            "collection": self._collection_name,
            "duration_ms": int((time.perf_counter() - start) * 1000),
            "results": [
                {"rank": i, "score": c.score, "file_path": c.file_path,
                 "chunk_id": c.chunk_id, "start_line": c.start_line}
                for i, c in enumerate(chunks)
            ],
        },
    )

This exact pattern — start_as_current_observation, as_type="span", per-result rank/score/file_path in the output metadata — already exists on search(), hybrid_search(), hybrid_search_rrf(), search_incidents(), and hybrid_search_incidents(). Every LLM call inside an agent is traced the same way, one level up, by the @trace_agent decorator (app/services/tracing.py:103-156), which wraps BaseAgent.run() as as_type="agent", with nested as_type="generation" and as_type="tool" spans auto-nested underneath via Langfuse’s contextvar-based tracing. A retrieval span that runs inside an agent’s tool call already lands nested under that agent’s trace automatically — no manual span-plumbing required, and no @observe decorator anywhere in this codebase, because the manual context-manager form is the pattern this project actually uses.

Which agents actually call retrieval, and where

An earlier draft claimed “the search call happens inside FixGenerationAgent and IncidentResponseAgent before the LLM call.” Checking the real call sites tells a different, more useful story:

• DiagnosisAgent (app/agents/diagnosis.py:573) — rag.hybrid_search(query, n_results=4, min_score=0.45). This is the pipeline’s primary RAG consumer, and it isn’t mentioned in the earlier draft at all.
• FixGenerationAgent (app/agents/fix_generation.py:2095) — self._rag.hybrid_search(...), but inside the self-critique pass, after the fix-generating call, feeding the critique call. Not “before the LLM call” — between two of them.
• IncidentResponseAgent (app/integrations/incident.py:294) — rag.search(...), exposed as a ReAct tool (search_codebase) the agent’s own LLM decides whether and when to invoke mid-loop. Not a fixed pre-step at all.
• Also real, also uncredited: CodeReviewAgent (app/agents/code_review.py:129) and CICDAgent (app/integrations/cicd.py:189) both call rag.search(), and incident_loop.py:465 calls rerank_incidents directly as part of the dedup gate covered earlier in this project’s blog history.

Getting this precise matters for tracing specifically: a span that assumes retrieval always happens once, always immediately before generation, will miss or mis-attribute what actually happens in a tool-call-driven agent, where retrieval can happen zero, one, or several times per run, interleaved with LLM calls rather than preceding a single one.

What was genuinely missing — and has since been built

The two pieces of the original proposal that weren’t already there turned into real, shipped code rather than staying a “you should build this” paragraph.

Index version attribution. RAGService now tracks a version per collection in a new index_metadata table (collection, version, updated_at). clear() bumps it — the real “a full rebuild just happened” signal (a chunking-strategy or embedding-model change, per Part 8), as distinct from the routine per-file incremental updates doc_chunk_registry already tracks on its own. It’s threaded into search()’s and hybrid_search()’s existing start_as_current_observation metadata, so a quality-drop trace can finally be correlated to which rebuild was live when it ran, without cross-referencing deploy timestamps by hand.

Building this caught a real bug before it shipped: the first version minted a version string at second granularity. Two clear() calls inside the same wall-clock second produced an identical “new” version — silently defeating the entire point of the feature for exactly the case (a fast retry, an automated rebuild loop) where you’d most want it to work. Fixed with microsecond precision; a regression test locks it in.

LLM-as-judge faithfulness/relevance scoring. app/services/rag_judge.py scores whether DiagnosisAgent’s generated root cause stayed faithful to the code it actually retrieved, and whether it addressed the incident. Sampled at 10% of real diagnoses, not every one — judging every run would double LLM cost for every incident, and this is explicitly a trend signal, not a per-case verdict, so a sample is enough. It is not a gate: DiagnosisAgent already has a deterministic grounding gate (submit_diagnosis rejects claims that don’t verify against the real repo, covered earlier in this project’s blog history) — this answers a different question about diagnoses that already passed that gate. Wired in via a new last_retrieved_chunks property on DiagnosisAgent (tracking every search_codebase tool call across one diagnose() run) and a new IncidentLoop._maybe_judge_diagnosis(), called right after diagnosis completes, wrapped in the same start_as_current_observation pattern as everything else in this post.

Building this one caught a more instructive bug: Claude Haiku reliably wraps its JSON output in a ```json fence, despite the prompt explicitly saying “Respond with ONLY a JSON object, no preamble.” A naive json.loads() on the raw response failed on every single real call during testing — not occasionally, every time. Left uncaught, this would have meant should_sample()’s 10% of judged diagnoses silently produced zero real scores, forever, with nothing anywhere surfacing that the judge itself was broken. The fix is a two-line regex strip before parsing. The lesson generalizes past this one prompt: “respond with only X” is a request, not a guarantee, and code that trusts it without verification finds out the hard way, at whatever sample rate you chose to trust it at.

Drilling into low-faithfulness traces typically surfaces one of three patterns, and only chunk-level attribution — which this project already has, via the existing start_as_current_observation metadata — lets you tell them apart:

Pattern	What you see in the trace	Root cause
Wrong document	Top chunks from a stale or deleted file	Index drift or missing deletion (Part 7’s “deleted files” gap)
Right document, wrong section	Top chunks from an adjacent function	Chunking boundary problem (Part 3)
Correct chunks, wrong answer	High faithfulness score, low relevance score	Generation problem — not retrieval

A postscript this post didn’t expect to need

Verifying these two features surfaced a third finding, entirely by accident, that fits the theme of this post better than anything planned for it. This project’s own CI has a hard regression gate on DiagnosisAgent: any change to it must replay a real golden dataset before merging — the same rigor this whole series has been arguing for. Shipping the changes above triggered that gate for real, for the first time in this series’ history (every prior post’s changes had path-filtered around it). The gate has a 180-minute timeout. The real replay didn’t finish in 180 minutes — it got through 8 of 56 cases before being cancelled, at roughly 10–25 minutes per case. The 8 that did complete were all correct (6 pass, 1 legitimate escalation, 1 cut off mid-run) — nothing about the two new features was actually broken. The gate’s own timeout budget was simply never tested against a real full run before, and didn’t survive contact with one.

That’s the closing argument of this whole post, not a footnote to it: a verification mechanism is only as trustworthy as the times it’s actually been exercised for real. This gate had been “confirmed live” in the sense that it existed, was wired into CI, and had auto-triggered correctly on the PR that introduced it. But in the six days between then and this post, diagnosis.py and base.py had zero other changes — meaning the gate had never once been asked to actually run its full replay for real since the day it shipped. The first genuine test of its own timeout budget happened to be this PR, and the budget didn’t survive it. That’s not a failure of this PR’s code — the 8 cases that did complete were all correct. It’s the offline-harness half of this post’s own opening argument, discovering a gap in itself.

---

This is the last post in this series so far. The chunking, retrieval, and rebuild pipeline built across Parts 1–8 now has both halves of verification in place: an offline recall harness for what’s known to be true before shipping, and production tracing — now with index-version attribution and sampled faithfulness scoring — for what’s actually happening after.