Document Registry — Keeping the Index Honest

Part 7 of the Code RAG series. Parts 1–6 built a retrieval pipeline that works. This post covers the piece that keeps it working over time: what happens when the code underneath the index changes, and why “just upsert it” isn’t the same as “keep the index correct.”

---

The problem: upsert doesn’t mean “replace”

A vector store’s upsert is insert-or-update by ID. That’s genuinely useful — re-running the indexer on a file whose content hasn’t changed is a safe no-op if the chunk IDs are stable. But upsert doesn’t know anything about other chunks that used to exist for the same file. It only ever touches the ID you hand it.

Picture what happens with pure upsert and no other bookkeeping, when moveAndRemoveFileFromS3 is refactored from one function into two:

Before:  moveAndRemoveFileFromS3  →  chunk ID a3f2 (lines 499–515)
After:   moveFile                  →  chunk ID b1c4 (lines 499–507)
         removeFile                →  chunk ID d9e7 (lines 509–515)

The new chunks b1c4 and d9e7 get upserted. Upsert alone never touches a3f2 — nothing told it that chunk belongs to a file that just changed shape. It stays in the collection, pointing at a function that no longer exists, and every future search over the codebase can still surface it. The LLM receives a ghost: a chunk describing code that was deleted, with no signal that it’s gone.

That’s the problem a document registry exists to solve — and it’s real code in this project, not a hypothetical. app/services/rag.py’s _index_file() already does this for every file it indexes.

---

The real registry

CREATE TABLE doc_chunk_registry (
    doc_id          TEXT NOT NULL,        -- relative file path
    chunk_vector_id TEXT NOT NULL,        -- chunk_id in the vector store
    content_hash    TEXT NOT NULL,        -- sha256 of file content
    collection      TEXT NOT NULL,        -- which vector collection this chunk lives in
    indexed_at      TEXT NOT NULL,
    status          TEXT NOT NULL DEFAULT 'active',  -- active | superseded
    INDEX idx_dcr_doc_id (doc_id),
    INDEX idx_dcr_chunk_vector_id (chunk_vector_id)
);

The re-index flow for a changed file, as actually implemented:

1. Hash check — sha256(new content) == registry hash for this doc_id? → skip, return 0
2. Delete old — fetch every active chunk ID registered for this doc_id, delete them all from the vector store
3. Mark stale — UPDATE status = 'superseded' WHERE doc_id = X AND status = 'active'
4. Chunk + embed — re-chunk the current content (function-boundary for JS/TS, header-based for Markdown, line-based otherwise — Part 3), embed, upsert
5. Register — insert new rows with status = 'active'

Step 2 is what actually prevents the ghost-chunk scenario above: it deletes every previously-active chunk for the whole file, unconditionally, before the new chunking even runs. It doesn’t need to know in advance whether a refactor produced the same chunk IDs or different ones — it clears the old set first, then computes and inserts the new one. File-granularity, not function-granularity, but that’s sufficient: any change inside a file, including a function splitting into two, is covered by clearing everything the registry knew about that file beforehand.

Steps 2–5 aren’t atomic across systems — the vector store and the registry’s database are separate — but the status column gives you observability into that: a row stuck active whose chunk ID no longer exists in the vector store is a detectable partial failure, not a silent one.

One correction to the chunk-ID scheme as originally described: it’s not a flat sha256("file_path:start_line") for everything. Line-based chunks use that form; function-boundary chunks are hashed as sha256("fn:file_path:start_line") and Markdown header chunks as sha256("md:file_path:start_line") — a type prefix, so a function chunk and a line-based fallback chunk starting at the same line number can never collide.

---

What this buys you — measured for real, not estimated

To check the “only changed files get re-embedded” claim wasn’t just true in theory, I ran it. Picked 10 real files already in the index — a mix of small config files and a couple of larger ones — and forced the registry to treat them as changed (corrupted their stored content hash, without touching a single byte of the actual source, so this is testing the registry mechanism in isolation):

create-post-fargate.js          1 chunk
config/authenticateToken.js     1 chunk
config/isSuperadmin.js          1 chunk
config/secretKey.js             1 chunk
README.md                       6 chunks
server.js                       6 chunks
config/city.js                  2 chunks
deploy-service.sh               4 chunks
config/validateMongoId.js       1 chunk
config/index.js                 11 chunks
                                 -----------
                                 34 chunks total

Then ran the real index_directory() across the full 457-file index and instrumented the actual embedding calls:

Real _embed() calls made:              10   (one per changed file — zero for the other 447)
Real total chunks embedded:            34
Files scanned but skipped (unchanged): 447 of 457  (97.8%)

That 97.8% figure is specific to this test, not a general constant — it’s a function of how many files out of the total actually changed, which depends entirely on the size of whatever change triggered the re-index. A 10-file PR skips nearly everything in a 457-file repo; a repo-wide reformat would skip almost nothing. There’s no fixed “60–80% of embed calls eliminated” number that holds across runs — that would need to be measured per-change, the same way this measurement was.

What is a fixed, structural property, not a per-run estimate: the number of embed calls scales with files-that-actually-changed, not with total-files-in-the-repo. That’s the real claim underneath the skip-on-unchanged behavior, and it holds regardless of which 10 files happen to change or how big the repo gets.

Stale chunks: confirmed zero for the tested files, via the registry’s own before/after state — every one of the 10 touched files had its previously-active chunk IDs deleted before the new ones were registered. (One limitation of this specific test worth being honest about: because I simulated “changed” via a hash mismatch rather than an actual content edit, the recomputed chunk IDs for these 10 files came out identical to the old ones — same file path, same line positions. That confirms the delete step runs cleanly on every changed file without erroring, but it doesn’t visually demonstrate an old ID vanishing while a different new one takes its place the way a real line-shifting refactor would. That specific behavior is verifiable directly from the code, though: the delete step reads the registry’s currently-active IDs and clears them before the new content is even chunked, so it can’t matter whether the freshly computed IDs happen to match the old ones or not.)

---

What this doesn’t solve

The registry handles file-level content changes correctly. Three real gaps remain, all still true of the shipped code, not just theoretical caveats:

Deleted files. If a file is removed from the repo, the indexer never visits a path that no longer exists, so its registry rows and vector-store chunks are never cleaned up. Confirmed directly: there’s no orphan-cleanup pass anywhere in rag.py. The fix would be a periodic full-run step that diffs registry doc_id values against the actual current file list and deletes anything registered but no longer present on disk.

Renamed files. A rename looks like a new file plus a deleted file to this system — the old path’s rows sit there as an undetected orphan (same failure mode as above) until that cleanup pass exists.

Collection drift. None of the four registry helper methods (_file_changed, _get_chunk_ids, _mark_superseded, _register_chunks) filter their queries by the collection column at all — confirmed by reading them directly. If more than one collection indexes overlapping doc_ids (which happens in this exact codebase — the same file paths get indexed into codebase, codebase_fn, and other experimental collections used elsewhere in this series), a hash-check or delete against one collection’s registry rows can silently affect bookkeeping that was meant for a different collection. This isn’t a hypothetical edge case here; it’s a real, currently-live gap in a system that already has multiple overlapping collections.

These are the edge cases that separate a demo index from a production one, and as of this writing, this system has solved the first problem (stale chunks within a file) and not yet the other three.

---

Next: Part 8 — incremental indexing via git diff: instead of re-scanning the full file tree on every run, read git diff HEAD~1 and update only the files that changed in the last commit.