Closing the Vocabulary Gap with LLM-Generated Descriptions
The error query went from not found to visible — sometimes. Sampling the same real model call eight times on the same function shows the fix works about half the time, not reliably. Here's the honest version of that story.
Part 4 of the Code RAG series. Part 3 fixed the chunking split by extracting function-boundary chunks with a real tree-sitter parser. This post tackles the second failure mode: the vocabulary gap — where the runtime error that fires in production appears nowhere in the source code.
The problem that chunking can’t fix
After Part 3, search results for moveAndRemoveFileFromS3 improved across the board. But one query was still completely broken:
Query: "S3 NoSuchKey missing key error"
Line-based: not found
Function-boundary: not found
The function calls s3.copyObject() and s3.deleteObject(). When copyObject is called with a source key that doesn’t exist in the bucket, the AWS SDK throws a NoSuchKey error. That’s the production incident.
But look at the source code:
async function moveAndRemoveFileFromS3(bucket, imageObj) {
try {
await s3.copyObject({
Bucket: bucket,
CopySource: `${bucket}/${imageObj.source}`,
Key: `${imageObj.destination}`,
}).promise();
await s3.deleteObject({ Bucket: bucket, Key: imageObj.source }).promise();
} catch (error) {
console.log("moveAndRemoveFileFromS3 error", error, bucket, imageObj);
}
}
Count the occurrences of “NoSuchKey” in that function: zero. The catch block logs error generically. The function has no idea what specific error it can produce — it just catches everything and logs it.
The vocabulary gap: the incident query uses NoSuchKey, the source code uses copyObject. Same concept — calling copyObject on a missing key produces NoSuchKey — but no shared token. The embedding model can’t bridge that gap through geometric proximity alone.
Why this is fundamentally different from the chunking problem
The chunking problem (Part 3) was about dilution: good information was present in the source but buried alongside unrelated code, weakening the embedding signal. The fix was structural — isolate the function.
The vocabulary gap is about absence: the information isn’t in the source at all. No amount of chunk isolation fixes it. The word “NoSuchKey” is simply not there.
Three approaches to close the gap:
Option A — Query expansion. Rewrite the incident query to include source-code vocabulary before searching. Requires knowing the mapping from error names to API calls, hardcoded or via an LLM call at query time.
Option B — Enrich at query time. Don’t change what’s indexed; use a cross-encoder re-ranker that can reason about whether a function could throw a given error even if the word isn’t there. (This is Part 6.)
Option C — Enrich the index at index time. When you index each function, append an LLM-generated description that includes the error names the function can throw. Embed code + description together. Now “NoSuchKey” might be in the chunk’s text.
Option C is the natural first step — a one-time cost at index time, no changes to the search query path. The “might” in that last sentence is the actual subject of this post.
The implementation
At index time, for each function, call Claude Haiku with this prompt (this is the real prompt, shipped in app/services/rag.py):
You are indexing JavaScript source code for semantic search. Write a single
sentence (max 40 words) describing this function. Include: what it does,
what external APIs or services it calls, and what errors or exceptions it
can throw (use the exact error names from those APIs, e.g. NoSuchKey,
NotFound, ValidationError). Do not include the function name. Output only
the sentence, no preamble.
This prompt is specific about including exact error names. The original assumption going into this post was that this instruction is what makes the model reliably say “throws NoSuchKey” instead of a vague “may throw errors.” Measuring it says otherwise.
What actually happens when you call it more than once
Here’s the real Haiku response, called against the actual moveAndRemoveFileFromS3 source, eight separate times across independent runs, same prompt, same model, same code:
1. [HIT] Moves a file in S3 by copying from source to destination then
deleting the original, using AWS S3 copyObject and deleteObject
APIs, catching NoSuchKey and other S3 errors.
2. [miss] Copies an S3 object to a new destination then deletes the
original using AWS S3 API, silently catching any errors without
rethrowing.
3. [miss] Copies an S3 object to a new destination then deletes the
original using AWS S3 API, silently catching any errors during
the process.
4. [HIT] Copies an S3 object to a new destination using the AWS SDK, then
deletes the original, silently catching and logging NoSuchKey and
other S3 errors.
5. [HIT] Copies an S3 object to a new location then deletes the original
using AWS S3 API, silently catching any errors including
NoSuchKey or NoSuchBucket exceptions.
6. [HIT] Copies an S3 object from source to destination using AWS SDK,
then deletes the original, catching and logging NoSuchKey or
other S3 errors without rethrowing.
7. [miss] Copies an S3 object to a new destination and deletes the source
using AWS SDK's copyObject and deleteObject methods, silently
catching any errors.
8. [miss] Copies an S3 object to a new destination using the AWS S3 API,
then deletes the original, silently catching any errors during
the process.
4 of 8 name the specific error. 4 of 8 don’t. That’s 50%, a coin flip.
Every one of these descriptions is accurate. Every one describes what the function does correctly, calls out the right APIs, and follows the prompt’s format. Only half happen to include the one specific word that makes the vocabulary-gap fix actually work for this query. The prompt’s explicit instruction to “use the exact error names” doesn’t make the model reliably comply — it makes the model capable of complying, which is a different thing.
This matters more than it sounds like it should, because the description gets generated exactly once, at index time, and then it’s frozen. Whichever of these possible sentences happens to get picked is what every future query lives with until the file is re-indexed.
What that non-determinism actually does to the ranking
To make this concrete rather than abstract: same function, same code, same embedding model — just swap in a “HIT” description for a “MISS” description and measure the same four queries.
Query Enriched with HIT desc Enriched with MISS desc
----------------------------------------------------------------------------------------
moveAndRemoveFileFromS3 rank 1 (0.6629) rank 1 (0.6617)
copy S3 object then delete source rank 1 (0.5878) rank 1 (0.5961)
S3 copyObject deleteObject bucket key rank 2 (0.6161) rank 2 (0.6025)
S3 NoSuchKey missing key error rank 8 (0.2901) not found
The first three queries don’t care — they’re not about the specific error, so the two descriptions perform almost identically. The fourth query is the entire point of this exercise, and it swings from “visible, barely, at rank 8” to “completely absent,” depending on which of several equally-plausible, equally-accurate sentences the model happened to generate the one time it was asked.
Rank 8 isn’t a win either — it’s still well outside a realistic top-3 or top-5 cutoff. So even in the better of the two real outcomes measured here, the fix doesn’t actually solve the problem on its own. It just makes the problem partially, unreliably visible.
Why even a hit doesn’t reach the top
The description that does include “NoSuchKey” still only reaches rank 8 at a low score (0.29). The query is about the error — what went wrong. The enriched chunk describes the function — what it does. Related concepts, not the same one.
The other results ranking higher are chunks that mention error handling, S3 operations, or key-not-found patterns more prominently in their own generated descriptions — other S3 functions in the same file, all competing on roughly the same vocabulary, with the actual target only weakly ahead of them even when it does have the right word.
What this means in practice
This post started from a different draft — one that ran the pipeline once, got a description that happened to name NoSuchKey, watched the rank move from “not found” to a specific number, and called it a fix. That’s not wrong, exactly. It’s an accurate description of one run. It’s just not a reliable description of what actually happens when you build this for real, because the thing being measured is a probability, not a fact, and reporting one sample of a probabilistic process as if it were deterministic is the same mistake this codebase’s own TriageAgent work already ran into and had to correct: temperature doesn’t have to be non-zero for a model’s output to vary in the ways that matter for retrieval — even a “small” wording change (whether a specific noun makes it into a 40-word sentence) is enough to flip whether a document is findable at all.
The honest framing: LLM-generated descriptions at index time are a real, cheap way to add vocabulary a bare code embedding doesn’t have. They are not a guarantee that the specific vocabulary a future query needs will show up. Whether it does was close to a literal coin flip for this function, on this prompt (4 of 8 real samples) — and that number is itself just an estimate from a small sample; it could be different for a different function, a different error, or a longer prompt.
Ways to push the odds, none of them a complete fix on their own:
- Longer description. A 3–5 sentence description with more room has more chances to mention the specific error, at higher token cost per function.
- Ask for the field explicitly, structured. Instead of one free-text sentence, request a structured
{summary, throws: [...]}shape — makes “did it name the error” a parseable yes/no instead of a hope, and makes it possible to detect and retry a miss automatically before it ever gets frozen into the index. - Hybrid search. A lexical boost means that on the runs where the description does say “NoSuchKey,” the exact token match counts for something beyond vector similarity alone — Part 5.
- Cross-encoder re-ranking. Doesn’t help the “MISS” case at all — if the word was never generated, there’s nothing for a re-ranker to find — but sharpens the “HIT” case considerably, as Part 6 measures directly.
Now real in production, not just this file
Since the draft of this post that measured a single run, this technique shipped for real: every function chunk across the full 454-file backend now gets a generated description as part of the actual indexing pipeline (app/services/rag.py, gated behind a settings flag), not just the one function used as this series’ running example. Real cost for the full codebase: $1.85 for roughly 4,200 functions. The non-determinism measured above doesn’t go away at that scale — it just becomes a property of the whole index instead of one function: some fraction of functions across the codebase will have descriptions that name their real failure modes precisely, and some fraction won’t, and which is which isn’t visible without checking.
Part 5 measures what a lexical/hybrid boost does for the case that does hit.