When to Re-chunk, Re-index, and Re-embed

Part 8 of the Code RAG series. This post covers the three most confused operations in a RAG pipeline — and why conflating them costs money and silently degrades retrieval quality.

---

The three operations, defined

Chunking is splitting a document into pieces. It’s a pure text operation — no API calls, no database writes. Input: a file. Output: a list of text segments with line ranges.

Embedding is converting a chunk of text into a vector. It requires an API call. It’s the expensive step — both in latency and cost. Input: text. Output: a vector (1536 numbers, for text-embedding-3-small, the model this pipeline uses).

Indexing is storing a vector (plus its metadata and original text) in the vector database. Input: a vector + metadata. Output: a row in ChromaDB or pgvector.

These three operations are often discussed as if they’re one thing (“re-index the codebase”). They’re not. Understanding when each is needed separately is how you avoid paying for thousands of embedding calls when 20 files changed.

One rule is absolute: if you re-chunk, you must re-embed. The vector stored in the database is computed from the chunk text. If the text changes and the vector doesn’t, they disagree — every similarity score is computed against a vector that doesn’t represent the stored text. Retrieval silently breaks. You can re-embed without re-chunking (a model upgrade). You can never re-chunk without re-embedding.

---

Re-indexing is not the same as re-embedding

“Re-index the codebase” is the phrase everyone uses, but it conflates two distinct operations.

Re-indexing means writing to the vector database — inserting new rows, deleting old ones, or updating metadata on existing rows. It is always the final step of the pipeline, but it can also happen without re-embedding:

• A file is deleted → delete its rows from the vector store. No embedding needed.
• A file is renamed → update the file_path metadata on its existing rows. The vector is still correct — the text didn’t change, only the path did.

Re-embedding means calling the embedding model to produce new vectors. It always requires re-indexing afterward (a vector that isn’t stored is useless), but re-indexing does not always require re-embedding.

Re-embedding always triggers re-indexing. Re-indexing does NOT always require re-embedding.

---

The three triggers

There are exactly three things that can make an index stale:

1. Document content changed — a file was edited, added, or deleted
2. Chunking strategy changed — you decided to split files differently
3. Embedding model changed — you switched to a different model

Each trigger has a different blast radius.

Trigger 1: Document content changed

A file was edited. What needs to happen?

File content changed
  ↓
Re-chunk that file        ← pure text operation, no API calls
  ↓
Re-embed the new chunks   ← API call per chunk, the expensive step
  ↓
Re-index: delete old chunk IDs from vector store
Re-index: insert new chunks + vectors into store
Re-index: update registry (mark old as superseded, register new)

Scope: one file. Every other file in the repo needs nothing. This is exactly what the document registry from Part 7 handles — the content-hash check catches it before any of the expensive steps run:

new_hash = hashlib.sha256(raw).hexdigest()
if not self._file_changed(relative, content_hash):
    return 0   # nothing changed, skip everything

What if the file was deleted? The indexer never visits a deleted file, so it never triggers the hash check — deletion has to be handled explicitly, by diffing the registry’s known files against what’s actually still on disk. Confirmed directly against the real code: this doesn’t exist yet. It’s the “deleted files” gap covered at the end of Part 7, not implemented, not a hypothetical.

Trigger 2: Chunking strategy changed

You’ve been chunking by fixed 50-line windows. You switch to function-boundary chunking (Part 3). What needs to happen?

Chunking strategy changed
  ↓
Re-chunk EVERY file       ← pure text operation, no API calls
  ↓
Re-embed EVERY chunk      ← API call per chunk, expensive
  ↓
Re-index: clear old collection  (chunk IDs from old strategy are meaningless)
Re-index: insert all new chunks + vectors
Re-index: rebuild registry from scratch

Scope: the entire corpus. Chunk IDs are derived from a hash of file_path + start_line (plus a type prefix — fn:/md:/none — since Part 3 and Part 7’s chunkers produce different chunk shapes for the same file). If line boundaries shift, which they do when the chunking strategy changes, the IDs change, and you cannot diff old against new by ID.

The right approach is alias-based deployment — build the new index in a parallel collection, validate it, then swap:

rag_v2 = RAGService(collection_name="codebase_v2")
await rag_v2.index_directory(CODEBASE_PATH)

results = await rag_v2.search("moveAndRemoveFileFromS3")
assert results[0].score > 0.70

# Swap — update the live collection name in config

RAGService genuinely supports collection_name as a constructor argument, so this pattern is directly usable against the real code as written — but it’s a recommended practice, not something this project has automated. There’s no existing script that does the build-validate-swap cycle; today, switching collections is a manual operation.

This isn’t a hypothetical failure mode — it already happened in this exact codebase. Before the tree-sitter chunker replaced the regex-based one (Part 3), the live codebase collection accumulated chunks from both strategies simultaneously: querying it mid-transition returned line-based chunks (441–490, 481–530, fixed 50-line windows) and function-boundary chunks (479–495, an old, since-drifted position for moveAndRemoveFileFromS3) for the same file, at the same time, because nothing had ever cleared the old ones. That’s Mistake 3, below — measured, not invented, from this project’s own history. Fixing it required a full clear() and rebuild, not an incremental patch.

Trigger 3: Embedding model changed

You switch from text-embedding-3-small to text-embedding-3-large. What needs to happen?

Embedding model changed
  ↓
Re-embed EVERY chunk      ← API call per chunk, expensive (no re-chunking needed)
  ↓
Re-index: clear old collection  (all vectors are wrong — different model's space)
Re-index: insert all chunks with new vectors

Scope: the entire corpus, even files that haven’t changed at all. The content is identical; the vectors are wrong anyway, because they were computed in a different geometric space. A query embedded with model B cannot meaningfully compare to documents embedded with model A — cosine distance between the two is comparing apples in one coordinate system to apples in an unrelated one.

existing_chunks = vector_store.all_items()
new_embeddings = await embed_all(
    [chunk.document for chunk in existing_chunks],
    model="text-embedding-3-large",
)
new_collection.upsert(zip(existing_chunks, new_embeddings))

Re-embedding this way reuses the already-stored chunk text — no need to re-read or re-chunk a single file, since the text itself didn’t change, only the model producing vectors from it.

A real gap worth naming plainly: this project doesn’t currently store which embedding model produced a chunk’s vector anywhere in its metadata. The recommended safeguard — stamp {"model": "text-embedding-3-small", ...} onto every chunk, and refuse to query a collection whose stored model doesn’t match the query-time model — is good practice and not implemented here. Today, if this pipeline’s embedding model were silently swapped without a full rebuild, nothing would raise an error. Queries would keep returning results, at scores that no longer mean anything, with no signal anywhere that something was wrong.

---

The decision matrix

What changed	Re-chunk?	Re-embed?	Re-index?	Scope
One file edited	Yes (that file)	Yes (new chunks)	Yes (delete old, insert new)	1 file
File deleted	No	No	Yes (delete rows) — not implemented	1 file
New file added	Yes	Yes	Yes (insert rows)	1 file
Chunking strategy changed	Yes (all files)	Yes (all chunks)	Yes (full rebuild)	Full corpus
Embedding model changed	No	Yes (all chunks)	Yes (full rebuild)	Full corpus
File renamed	No	No	Yes (update metadata only) — not implemented	1 file
Nothing changed	No	No	No	Skip

---

The cost hierarchy — real numbers, not round ones

Read file from disk:                           ~0 cost
Hash the content:                              ~0 cost
Re-chunk a file:                               ~0 cost   (pure CPU, no API)
Re-index: delete old chunks from vector store  ~0 cost   (one delete call)
Re-index: insert new chunks into vector store  ~0 cost   (one write call)
Re-index: update registry row                  ~0 cost   (one SQL write)

Re-embed one chunk (real corpus average, 345 tokens):  ~$0.0000069
Re-embed the full real corpus (4,883 chunks):          ~$0.03

Pulled directly from the real indexed collection rather than assumed: 4,883 chunks, averaging 345 tokens each (not the 512 you might guess), at text-embedding-3-small’s real current rate of $0.02 per million tokens. Full-corpus re-embed cost: about 3 cents, not the $0.11 a naive 512-tokens-per-chunk estimate would suggest — a good reminder that “chunks are short functions and small config files,” not “chunks are all near the 50-line maximum,” and average chunk size matters for a real cost estimate, not just chunk count.

Re-chunking is free. Re-indexing is free. Re-embedding is the only expensive operation, and even at scale it’s a small number for a codebase this size.

The hash check converts re-embedding from O(corpus size) to O(changed files). This isn’t a hypothetical multiplier — Part 7 measured it directly: touching 10 real files out of a 457-file corpus produced exactly 10 embed calls, not 457. That’s a real, measured 45.7x reduction for that specific test, not a fixed constant — the actual ratio scales with however many files a given change touches, same as the dollar cost above.

The registry is the mechanism that makes deletion possible without a full rebuild. Without it, there’s no way to know which chunk IDs to delete when a file changes — the registry stores that mapping cheaply at index time so deletion at update time is a fast, indexed lookup, not a full-collection scan.

The chain of reasoning: re-embedding costs money → hash-check files to skip unchanged ones → need the hash stored somewhere → registry → the registry also stores chunk IDs → enables correct deletion → stale vectors don’t accumulate → retrieval stays honest. Every piece exists because re-embedding costs money, even when that cost is, in absolute terms, small.

---

Common mistakes

Mistake 1: Re-embedding on every index run. No hash check, so every run re-embeds every file regardless of whether it changed. Costs money every time, with zero improvement in retrieval quality.

Mistake 2: Re-embedding when only metadata changed. A file rename, a path restructure. The vector for the chunk is still correct — only the file_path field needs updating. No embed call required. (Also not currently implemented in this project — see Trigger 1’s file-rename gap above.)

Mistake 3: Not deleting old chunks when chunking strategy changes. This is the one that actually happened here, not a cautionary hypothetical. Switching to function-boundary chunking changed every chunk’s ID (different start_line values, different hash prefix). The old line-based chunks were never explicitly cleared before the switch, and the live codebase collection carried both simultaneously for a while — confirmed directly by querying it and finding both a function-boundary chunk at a since-drifted line range and multiple 50-line-window chunks for the identical file, coexisting. Searches during that period returned a mix of both strategies. Fixing it required treating it as Trigger 2 above: a full clear() and rebuild, not an incremental patch.

Mistake 4: Not rebuilding when the embedding model changes. The silent worst case. The index looks fine. Queries return results. The scores are meaningless, because a query embedded with the new model is being compared to documents embedded with the old one. You won’t catch this without an eval harness checking against known correct answers — which is exactly why this series has been running the same four queries against every change, not just eyeballing whether results “look reasonable.”

---

The mental model

Three questions, in order:

1. Did the text change? No → skip. Yes → re-chunk + re-embed that file, delete its old chunks.
2. Did the chunking strategy change? Yes → full rebuild (all files, all chunks). Alias-based deployment, not an in-place patch.
3. Did the embedding model change? Yes → full re-embed (all chunks, no re-chunk needed). Same alias-based approach.

Answer these three questions before any index operation and you avoid paying for unnecessary embed calls — and, per this project’s own history, avoid actually shipping a collection with two incompatible chunking strategies silently coexisting in it.

---

Next: Part 9 — incremental indexing via git diff: instead of re-scanning the full file tree on every run, read git diff HEAD~1 and update only the files that changed in the last commit.