Search Quality — Two Failure Modes and Why They're Different

Part of a series on building production RAG over a real JavaScript codebase. Part 1 showed how 50-line chunks split functions across boundaries — and, measuring it again for this post, that whether a given function actually gets split is a moving target, not a fixed property of the function. This post measures what a chunking split costs in search quality, using a function that’s currently split — and finds a second, deeper failure that chunking alone can’t fix, using the function this whole series is actually about.

---

Setup

The incident this series is built around: moveAndRemoveFileFromS3, at the center of a real S3 NoSuchKey failure. It lives at line 499 of routes/services/image.js and does exactly two things — copy an S3 object to a new key, then delete the original.

async function moveAndRemoveFileFromS3(bucket, imageObj) {
  try {
    if (!imageObj.source || !imageObj.destination) return;
    if (imageObj.source === imageObj.destination) return;

    await s3
      .copyObject({
        Bucket: bucket,
        CopySource: `${bucket}/${imageObj.source}`,
        Key: `${imageObj.destination}`,
      })
      .promise();
    await s3.deleteObject({ Bucket: bucket, Key: imageObj.source }).promise();
  } catch (error) {
    console.log("moveAndRemoveFileFromS3 error", error, bucket, imageObj);
  }
}

Here’s a wrinkle worth stating plainly rather than hiding: as of Part 1, this function no longer gets split by the 50-line chunker. It used to — at an earlier point in the file’s history it lived at line 479 and was cut across two chunks the same way the example below is. Since then, unrelated code was added earlier in the file, and it now lands entirely inside one 50-line window (481–530). Its own content never changed; only its position did.

That’s inconvenient for demonstrating the chunking-split failure mode with this exact function today, but it’s not a problem for the series — it’s actually the cleanest illustration of why fixed-size chunking is unreliable in the first place. So this post measures the chunking-split failure mode with a function that is currently split — UploadOriginaltoS3, a 17-line function later in the same file — and keeps moveAndRemoveFileFromS3 for the second failure mode, which has nothing to do with chunk boundaries and is entirely unaffected by where the function happens to sit today.

// UploadOriginaltoS3 — currently split across two 50-line chunks (281–330, 321–370)
async function UploadOriginaltoS3(bucket, baseFileKey, compressFileKey) {
  try {
    const signUrl = s3.getSignedUrl("getObject", { Bucket: bucket, Key: baseFileKey, Expires: 18000 });
    const image = await fetch(signUrl);
    let buffer = await image.buffer();
    return s3
      .putObject({ Bucket: bucket, Key: compressFileKey, Body: buffer, ACL: "public-read" })
      .promise();
  } catch (exception) {
    console.error("compressAndUploadtoS3", exception, bucket, baseFileKey, compressFileKey);
  }
}

I ran four queries against the full line-based index (27 chunks from this one file, indexed in isolation for a clean measurement) and tracked both functions.

---

The four queries

Query                                     moveAndRemoveFileFromS3        UploadOriginaltoS3
                                           (1 chunk, 481–530)             (2 chunks: 281–330 / 321–370)
--------------------------------------------------------------------------------------------------
moveAndRemoveFileFromS3                   rank 3   (0.4927)              rank 9/10  (0.4192 / 0.4189)
copy S3 object then delete source         rank 2   (0.4509)              rank 7     (0.4090)
S3 copyObject deleteObject bucket key     not in top 10                  rank 5     (0.4572)
S3 NoSuchKey missing key error            not in top 10                  rank 7/10  (0.2656 / 0.2616)

Two distinct stories.

---

Failure mode 1: the chunking split

UploadOriginaltoS3 — the currently-split function — is buried. For its own name, both of its fragments land at rank 9 and 10, barely inside the top 10 at all, with near-identical mediocre scores (0.419, 0.419) because each fragment is a coin flip between “has the declaration, missing the catch block” and “has the catch block, missing the declaration.” Neither fragment is a complete, coherent description of what the function does, so neither scores well against a query that’s about the function.

moveAndRemoveFileFromS3, sitting intact in one chunk today, does noticeably better on the same queries — rank 2–3 instead of rank 5–10 — despite being a similar size, similar shape, similar S3-copy-then-delete pattern. The only structural difference between them right now is whether they happen to fit inside a single 50-line window. That’s the chunking-split failure mode in one comparison: two functions, same file, same general shape, and a large gap in retrievability that has nothing to do with what either function does.

The practical consequence for UploadOriginaltoS3: an agent doing code search for it gets back two incomplete fragments, neither of which is a coherent implementation. It never sees the full function as one thing.

---

Failure mode 2: the vocabulary gap

moveAndRemoveFileFromS3 does reasonably well on three of the four queries. The fourth — S3 NoSuchKey missing key error — doesn’t appear in the top 10 at all, chunked whole or not.

This is not a chunking problem. The chunk is intact, undiluted, exactly the function. It still fails.

The reason: the runtime error has no textual trace in the source code. The incident that triggered the alert was copyObject throwing NoSuchKey when the source key didn’t exist. But the source code says:

await s3.copyObject({
  Bucket: bucket,
  CopySource: `${bucket}/${imageObj.source}`,
  Key: `${imageObj.destination}`,
}).promise();

There is no NoSuchKey. No comment like // throws NoSuchKey if source doesn't exist. No error code in the catch block — just console.log("moveAndRemoveFileFromS3 error", error, ...). The source and the error live in completely different vocabularies.

Compare this to incident RAG, where the error text itself gets indexed. Querying a real indexed production incident with its own (near-exact) description scores 0.86 against itself — and the next-closest, genuinely unrelated incident scores 0.36. That’s the kind of gap you get when the query and the document are written in the same vocabulary on purpose. Code RAG doesn’t have that advantage: the function call and the exception it can throw are two different documents with no shared tokens, no matter how the chunk is drawn.

---

Why this matters more than the chunking split

The chunking split is a fixable engineering problem — chunk by function boundary instead of line count, and UploadOriginaltoS3 stops being two incoherent fragments. Part 3 does exactly that.

The vocabulary gap is a harder problem. It’s not about how you chunk. It’s about what information you have. The source code describes what the function does. The error query describes what the function can fail with. Those are different things and they don’t share vocabulary unless a developer explicitly wrote comments linking them.

Three approaches to close the gap:

1. Enrich the indexed text at index time. For each function, generate a description: “this function calls copyObject and deleteObject; it can throw NoSuchKey if the source key doesn’t exist.” Store that alongside the raw source. Now the chunk’s embedded text spans both vocabularies. Cost: one LLM call per function at index time. Quality depends on the LLM’s knowledge of the AWS SDK — and, as Part 4 covers, on the LLM actually choosing to mention the specific error every time, which it doesn’t.
2. Query expansion at search time. Before searching, rewrite the incident query to include likely source vocabulary. “S3 NoSuchKey missing key” → “copyObject deleteObject S3 key missing”. Cost: one LLM call per query. Requires knowing what SDK calls can throw what errors.
3. Accept the gap and use the GitHub API for the last mile. Incident RAG already retrieves the right past incident (what failed and why). Code RAG retrieves semantically related source (what code is involved). If neither retrieves the exact function, the agent can still find it via GitHub’s search_code API with the exact function name. Code RAG is a first-pass filter, not the only path.

Option 1 is covered starting in Part 4, with hybrid search (Part 5) and cross-encoder re-ranking (Part 6) layered on top of it.

---

Score calibration for code RAG

The scores in this measurement are worth examining independently of rankings:

Best result, either function:      0.49  (moveAndRemoveFileFromS3, function name query)
Best result, natural language:     0.45  (moveAndRemoveFileFromS3, copy/delete description)
Best result, incident vocabulary:  not found in top 10

Compare this to incident RAG, where a real production incident matched against its own (near-identical) text scores 0.86. Code search scores here are meaningfully lower across the board.

This is expected. Incident text was written in natural language about concepts the embedding model understands well. Source code is written in a mix of identifiers, API calls, and language syntax — it maps more weakly to the embedding model’s training distribution, which is mostly natural-language text.

The practical implication: don’t port a min_score threshold calibrated for incident RAG onto code RAG. A score of 0.45–0.50 for code search here is a genuinely good match. Calibrate separately, and expect the ceiling to be lower.

---

Summary

Failure	Root cause	Fixable by better chunking?
Function fragmented across chunks (UploadOriginaltoS3)	Fixed line-count windows don’t respect function boundaries	Yes — chunk by function boundary
Incident error vocabulary absent from source (moveAndRemoveFileFromS3 + NoSuchKey)	Runtime errors aren’t annotated in source	No — requires richer indexed text or query expansion

The chunking split means a query for a fragmented function returns an incomplete, low-scoring result. The vocabulary gap means a query for the incident error returns nothing at all, even when the target chunk is whole and undiluted. Both are real, but they’re different problems with different solutions — and, as this post’s own re-measurement shows, which functions currently suffer from the first one can change under you without any of your code changing.

---

Next: Part 3 — function-boundary chunking with a real tree-sitter parser, applied across the whole file, not just the two examples in this post.