Cross-Encoder Re-Ranking — A Real Negative Result

Part 6 of the Code RAG series. Part 5 found that hybrid search pushes the vocabulary-gap query’s target function from rank 11 to rank 4 — a real improvement, conditional on Part 4’s coin flip landing on the description that names the specific error. This post applies cross-encoder re-ranking, the same two-stage pattern used for incident search earlier in this series. The expected result, going in, was that reading the query and each candidate together would do better than a fixed lexical weight at picking the right function out of a crowded field. That’s not what happened.

---

A correction before anything else

Cross-encoder re-ranking for code search did not exist anywhere in this codebase before the work behind this post. Not in production, not in an exercise script. rerank_incidents() existed for incident search, but the equivalent for code — rerank_functions() — had to be built from scratch, and sentence-transformers had to be added as an installable dependency for the first time (it was already imported, behind a bare try/except ImportError, by code that had apparently never actually run in a real environment). This post’s numbers are the first real measurement of this technique that has ever existed for this part of the system.

---

Where we left off

Query: "S3 NoSuchKey missing key error"

Part 4 (vector only), HIT description:    rank 11   (0.29)
Part 5 (hybrid), HIT description:         rank 4    (0.44)
Both parts, MISS description:             not found

The question this post asks: does reading the query and each candidate jointly — the whole premise of a cross-encoder — do better than hybrid search’s fixed lexical weight at pulling the actual target ahead of the other four legitimate-looking answers from Part 5?

---

The architecture

Real production code, app/services/rag.py, mirroring rerank_incidents()’s exact pattern:

async def rerank_functions(self, query, n_results=3, candidate_pool=20):
    candidates = await self.search(query, n_results=candidate_pool, min_score=0.0)
    if not candidates:
        return []
    if not hasattr(self, "_cross_encoder"):
        self._cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
    pairs = [(query, c.content) for c in candidates]
    ce_scores = self._cross_encoder.predict(pairs)
    for c, score in zip(candidates, ce_scores):
        c.score = round(float(score), 4)
    return sorted(candidates, key=lambda c: c.score, reverse=True)[:n_results]

Two stages: vector search fetches a wide candidate pool (20, unconditional, no score floor — recall), then the cross-encoder reads the query and each of those 20 candidates’ full text jointly and re-scores them for actual relevance (precision). This is architecturally identical to the incident-search cross-encoder covered earlier in this series.

---

What it does for the queries it’s good at

Query                                     HIT description         MISS description
------------------------------------------------------------------------------------
moveAndRemoveFileFromS3                   rank 1   (ce=7.73)       rank 1   (ce=7.73)
copy S3 object then delete source         rank 1   (ce=7.11)       rank 1   (ce=6.46)
S3 copyObject deleteObject bucket key     rank 1   (ce=7.71)       rank 1   (ce=7.75)

This is the cross-encoder working exactly as advertised. Compare the score spread here to hybrid search’s: hybrid scores for these queries clustered around 0.6–0.8; cross-encoder scores span roughly 6 to 8, with a real, wide margin over whatever’s in second place. For queries with one clear right answer, a cross-encoder that reads the full text of query and candidate together is a much more confident, better-separated signal than cosine similarity plus a lexical tiebreaker.

---

What it does for the query it was actually built for

Query: "S3 NoSuchKey missing key error"

                    Vector only   Hybrid    Cross-encoder
HIT description     rank 11       rank 4    rank 11  (ce=0.08)
MISS description     not found    not found  not found

Cross-encoder re-ranking makes the HIT case worse than hybrid search, not better. And the MISS case is exactly as unreachable as it always was — for a specific, structural reason worth being precise about, because it’s not the same reason as Part 5.

Why MISS never had a chance

rerank_functions()’s first stage is a plain vector search with no score floor, fetching the top 20 candidates by cosine similarity. In the MISS-description index, the target function doesn’t make that list at all — it’s not in the top 20 by vector similarity, period. The cross-encoder never sees it, because the cross-encoder only ever re-scores whatever the first stage already surfaced. This is the honest limit of any two-stage retrieval architecture: the second stage can only improve precision within the recall the first stage already achieved. If recall excludes the answer, no amount of re-ranking sophistication downstream recovers it.

Why HIT gets worse, not better

Here the target does reach the candidate pool (rank 11 by vector score, same as Part 4’s raw number). The cross-encoder reads all 20 candidates against the query and produces this ranking:

rank 1   ce=3.97   getBufferFromBucket
rank 2   ce=3.54   getImagesDimensions
rank 3   ce=2.93   convertToJPEGFromBuffer
rank 4   ce=2.69   UploadOriginaltoS3
rank 5   ce=1.45   fixFailedCompressedImages
rank 6   ce=1.25   handleCompression
rank 7   ce=0.79   compressAndUploadtoS3
rank 8   ce=0.78   generateThumbnailAndUploadtoS3
rank 9   ce=0.59   s3ObjectExists
rank 10  ce=0.49   saveImageToOriginal
rank 11  ce=0.08   moveAndRemoveFileFromS3   <-- TARGET

Ten other real functions outrank the actual target. This isn’t the cross-encoder malfunctioning — every one of these ten genuinely deals with S3 operations, buffers, image processing, or error handling, and the cross-encoder is doing what it’s designed to do: judging semantic relevance between the query text and each candidate’s full text. What it’s not doing, and structurally can’t do, is know which function is ground truth for a specific historical incident. “Relevant to a query about S3 key errors” and “is the function that actually threw this specific error in production” are different questions, and a general-purpose cross-encoder trained on query-passage relevance answers the first one, not the second.

The vector-only and hybrid stages happened to rank the target more competitively (rank 11, then rank 4) partly by chance — the specific words in its generated description lined up reasonably well with the query’s words. The cross-encoder, reading full text instead of comparing bag-of-words overlap, picked up on other functions’ more central, more detailed treatment of S3 error handling and ranked them higher. That’s a real, defensible judgment about the text it was given. It’s just not the answer this series needed.

---

The honest lesson

Two-stage retrieval (bi-encoder recall, cross-encoder precision) is a real, well-established pattern, and it’s not wrong to reach for it — it’s the right architecture for the general problem of “find the most relevant document to this query.” What this measurement shows is narrower and more useful than “the pattern doesn’t work”: it shows that “most relevant to the query” and “the specific artifact behind this specific incident” are not the same target, and nothing in this architecture is aimed at the second one. A cross-encoder sharpens relevance judgments; it doesn’t know which of several relevant functions actually produced the error you’re investigating. For incident-diagnosis retrieval specifically, that’s a real gap — one this post doesn’t have a fix for, only a precise description of.

The previous draft of this post, before it was measured for real, claimed the opposite: a clean win, rank 3 to rank 1, a wide score gap in the target’s favor. That version of events required a specific measurement that never actually happened. It read as intuitively correct — cross-encoders are supposed to be strictly better than lexical heuristics for hard queries — and that intuition is usually right. It just wasn’t right here, and the only way to know that was to actually build the thing and run it, which is the entire reason this reconstruction exists.

---

Summary

	Vector only	Hybrid	Cross-encoder
Function name query	rank 1	rank 1, higher score	rank 1, much wider margin
Natural language query	rank 1	rank 1, higher score	rank 1, much wider margin
API call names query	rank 2	rank 1	rank 1, much wider margin
Vocabulary-gap query, HIT description	rank 11	rank 4	rank 11 — worse than hybrid
Vocabulary-gap query, MISS description	not found	not found	not found — never reaches the candidate pool

For unambiguous queries, cross-encoder re-ranking is a clear, measured improvement — wider score margins, more confident separation between the right answer and everything else. For the query this whole series is built around, it isn’t, and the reason is structural rather than incidental: the vector stage gates what the cross-encoder can ever see, and among the candidates it does see, the cross-encoder optimizes for a genuinely different question than “which function caused this incident.”

The series started by measuring how much a 50-line window costs you compared to a real function boundary — a clean, large, reproducible win. It ends by measuring a much more sophisticated technique that, for the actual case that motivated building it, makes things worse. Both are real findings. Only one of them was the one this series expected to report.

---

Next: Part 7 — the Document Registry: what happens when the codebase changes and the index goes stale.