Hybrid Search — Rescuing the Coin Flip, Not Fixing It

Part 5 of the Code RAG series. Part 4 found that LLM-generated function descriptions close the vocabulary gap only probabilistically — sampling the real model call eight times on the same function, only half named the specific error (NoSuchKey) it can throw. This post applies hybrid search — the same technique used in the Incident RAG series — and finds it has a real, large effect on one of those two outcomes, and zero effect on the other.

---

Where we left off

Part 4’s honest finding: whether moveAndRemoveFileFromS3’s generated description contains the word “NoSuchKey” is close to a coin flip. Two real, equally likely outcomes of the exact same pipeline:

Query: "S3 NoSuchKey missing key error"

HIT run  (description names NoSuchKey):     rank 8,       score 0.2901
MISS run (description doesn't):             not found in top 10

Rank 8 is still well outside a usable cutoff even in the better case. The question this post answers: does hybrid search fix that, and does it help the worse case at all?

---

The hybrid formula

Real production code, app/services/rag.py, already wired into DiagnosisAgent and FixGenerationAgent’s retrieval — not a standalone exercise:

def hybrid_score(vector: float, document: str, query: str, alpha: float = 0.7) -> float:
    tokens = set(query.lower().split())
    matched = sum(1 for t in tokens if t in document.lower())
    lexical = matched / len(tokens) if tokens else 0.0
    return alpha * vector + (1 - alpha) * lexical

alpha = 0.7 — semantic similarity is the primary signal, lexical token overlap is a tiebreaker/boost. For the query “S3 NoSuchKey missing key error,” the query tokens are {s3, nosuchkey, missing, key, error} — five tokens. Whether “nosuchkey” is one of the tokens actually found in a chunk’s text depends entirely on which of Part 4’s eight possible descriptions got frozen into the index.

---

What hybrid search actually does to each real outcome

Ran the real hybrid_search() method — not a reimplementation — against two real indexes: one built with a description that names NoSuchKey, one built with a description that doesn’t.

Query                                    HIT description         MISS description
------------------------------------------------------------------------------------
moveAndRemoveFileFromS3                  rank 1   (0.7640)        rank 1   (0.7632)
copy S3 object then delete source        rank 1   (0.7115)        rank 1   (0.6670)
S3 copyObject deleteObject bucket key    rank 1   (0.7313)        rank 1   (0.7218)
S3 NoSuchKey missing key error           rank 2   (0.4431)        not in top 10

The first three queries improve nicely in both cases — hybrid search’s lexical boost rewards exact identifier overlap (copyObject, deleteObject, the function’s own name) regardless of which description variant got indexed, since those tokens are in the code itself, not just the generated text.

The fourth query is where the two real outcomes diverge completely. HIT: rank 8 → rank 2, a genuinely large, real improvement — better than a single earlier measurement of this same technique suggested. MISS: not found → still not found. Hybrid search’s lexical term has nothing to work with, because the token “nosuchkey” was never written into the document in the first place. You cannot get lexical credit for a word that isn’t there.

This is the sentence that matters most in this post: hybrid search rescues the coin flip’s “heads” outcome. It does nothing whatsoever for “tails.” It is not a fix for Part 4’s non-determinism — it’s an amplifier that only has something to amplify half the time.

---

Why HIT reaches rank 2, not rank 1

The top 5 results for the vocabulary-gap query, HIT description, real hybrid scores:

rank 1  hybrid=0.4632  s3ObjectExists
rank 2  hybrid=0.4431  moveAndRemoveFileFromS3   <-- TARGET
rank 3  hybrid=0.4363  checkFileExists
rank 4  hybrid=0.4144  fixTiffImage
rank 5  hybrid=0.4113  removeFile

s3ObjectExists edges it out — legitimately. That function’s own source code explicitly checks if (err.code === 'NotFound'), which is real, relevant vocabulary for a “missing key” query, arguably even more directly on-topic than moveAndRemoveFileFromS3’s generated description mentioning NoSuchKey in passing. This isn’t hybrid search misfiring; two functions in the same file are both genuinely relevant to “what happens when an S3 key is missing,” and the ranking between them is close and reasonable.

The gap between rank 1 and rank 2 here (0.4632 vs 0.4431) is real and small — both are legitimate answers to the query, not one right answer and one noise result.

---

What this means for the fix as a whole

Chaining Part 4 and Part 5 together, the honest end-to-end picture for the vocabulary-gap query is:

                          Part 4 (enriched, vector only)   Part 5 (enriched + hybrid)
HIT description (~50%)   rank 8    (0.29)                 rank 2    (0.44)
MISS description (~50%)  not found                        not found

Roughly half the time, this two-part fix (LLM description + hybrid search) works well — the target function reaches rank 2, solidly inside any reasonable cutoff. The other half of the time, it doesn’t work at all, and no amount of hybrid tuning changes that, because the problem at that point isn’t retrieval — it’s that the vocabulary was never generated to begin with.

That reframes what Part 4 and Part 5 actually are. They’re not two sequential fixes that compound into a reliable solution. Part 5 is a real, substantial improvement conditional on Part 4 already having worked. Neither post, on its own or together, makes the vocabulary-gap query reliably findable — they make it conditionally findable, and the condition is decided once, silently, at index time, by a model call nobody checks the output of.

Part 6 asks whether cross-encoder re-ranking changes this picture — spoiler: it sharpens the HIT case further, the same way hybrid search did, and is exactly as unable to help the MISS case, for exactly the same reason.

---

Why alpha = 0.7 works here

With alpha = 0.7:

• A chunk with a perfect vector score (1.0) and zero lexical overlap scores 0.70
• A chunk with zero vector score and perfect lexical overlap scores 0.30
• The HIT-description target (vector 0.28, lexical 0.80 — 4 of 5 query tokens present) scores 0.44

The lexical bonus is real and meaningful, but it can’t override a genuinely strong vector match elsewhere — semantic relevance stays primary, lexical overlap is a tiebreaker. That’s also exactly why it can’t manufacture a token that isn’t in the text: 0.7 × (anything) + 0.3 × 0 is still capped by the vector term alone, and the vector term for a MISS-description chunk was never going to rank near the top on this query by itself either.

---

Summary

	Part 4 (vector only)	Part 5 (hybrid)
Function name query	rank 1	rank 1, higher score
Natural language query	rank 1	rank 1, higher score
API call names query	rank 2	rank 1
Vocabulary-gap query, HIT description	rank 8	rank 2
Vocabulary-gap query, MISS description	not found	still not found

Hybrid search is a real, well-targeted improvement for exact-identifier and exact-error-name queries. It is not, and structurally cannot be, a fix for the case where the vocabulary was never indexed in the first place. That’s a different problem, and it doesn’t have a query-time solution — it needs either a more reliable generation step (structured output, retries, a longer description) or acceptance that some fraction of functions in any codebase indexed this way will stay hard to find on their specific failure modes until they’re re-indexed with a description that happens to name them.

---

Next: Part 6 — cross-encoder re-ranking, and whether reading the query and the candidate together does any better than a fixed lexical weight at telling moveAndRemoveFileFromS3 apart from s3ObjectExists — two functions that are both, honestly, relevant answers to the same query.