RAG Finds the Candidate. The Live Store Confirms the Truth.

When I added semantic deduplication to my incident pipeline, I made a mistake that a production failure exposed: I trusted the vector index to tell me the current state of an incident.

It couldn’t. It never could. That’s not what it’s for.

Here’s what I learned about the difference between a search index and a source of truth — and why confusing them is a category error, not just a bug.

---

The Setup

My incident pipeline runs when a production error is detected on ECS Fargate. Before doing any expensive work — triage, diagnosis, fix generation — it checks whether the same error is already being handled. If there’s an open incident with an open PR, drop the duplicate and move on.

The deduplication check used RAG to find semantically similar past incidents. In production, that’s pgvector inside Postgres. The search returns each matching incident along with metadata — status, pr_url — that was stored when the incident was indexed. The code treated that metadata as current:

similar = await self._rag.search_incidents(query, min_score=0.90)
for s in similar:
    # s["status"] and s["pr_url"] came from the vector index —
    # stored at index time, not updated since
    if s["status"] not in terminal_states and s["pr_url"]:
        return  # drop the duplicate

If s["status"] says the incident is still active and there’s a PR open, drop the incoming event. The problem: s["status"] is a snapshot from when the incident was first indexed. If the incident has changed state since — resolved, failed, merged — the vector index has no idea. You’re making a blocking decision based on stale data.

---

What the Vector Index Actually Is

pgvector stores embedding vectors alongside metadata you provide at index time. You give it text, it converts that text to a vector and stores it. Later, you query it, it finds the stored vectors most similar to your query and returns them along with whatever metadata you attached.

It is optimized for one thing: finding things that are semantically similar to other things. It is extraordinarily good at this.

It is not a live database. It doesn’t watch your incidents and update itself when they change state. It stores exactly what you gave it at index time, and that’s what it returns — until you explicitly re-index.

My code indexed incidents like this:

metadatas=[{
    "incident_id": incident.id,
    "status": incident.status.value,   # snapshot at index time
    "pr_url": incident.pr_url or "",   # snapshot at index time
}]

And only called index_incident() in two places: when an incident was created, and when it resolved.

So for the entire active lifetime of an incident — TRIAGING → FIXING → REVIEWING → AWAITING_APPROVAL → AWAITING_REFIX_APPROVAL — pgvector held a frozen snapshot from the moment the incident was born. Five state transitions invisible to it.

---

What the Live Store Actually Is

The live store is Postgres — the same database that pgvector runs inside, but queried directly. Every state transition writes to it immediately:

incident_store.update(incident_id, status=IncidentStatus.REVIEWING)
incident_store.update(incident_id, pr_url="https://github.com/...")

At any point in time, incident_store.get(incident_id) returns the real current state — not a snapshot, not a cache, the actual truth right now.

When the duplicate error arrived and the incident was at AWAITING_REFIX_APPROVAL, Postgres knew that immediately. The pgvector metadata still had TRIAGING from four state transitions ago.

---

The Failure

The same TypeError: Cannot read properties of undefined (reading 'publish_decision') arrived again on the production cluster. An open incident already existed with an open PR. The pipeline should have dropped it.

Instead it ran the full pipeline — triage, diagnosis, fix generation, a second PR for a bug that already had a PR open.

Tracing through why:

Layer 1 (string match)     → missed due to whitespace difference in log format
Layer 2 (SQL lookup)       → found a resolved incident, set regression context
Layer 3 (RAG)              → gated behind Layer 2, never ran

Even if Layer 3 had run, it would have read s["status"] from pgvector metadata — which showed TRIAGING, not AWAITING_REFIX_APPROVAL. The blocking check would have correctly identified this as a non-terminal state and dropped the event. But only accidentally — the status it read was wrong, it just happened to be wrong in the right direction.

That’s a broken clock being right twice a day. The next incident might be in a state where the stale status was terminal-looking, and a real duplicate would slip through.

---

The Fix

Two lines changed:

# Before — trusting pgvector metadata
if s["status"] not in terminal_states and s["pr_url"]:
    return

# After — using pgvector only to find the candidate
live = incident_store.get(s["incident_id"])
if live and live.status not in terminal_states and live.pr_url:
    return

pgvector still does the search. It’s excellent at finding semantically similar incidents — that part works perfectly. But the moment we have a candidate incident ID, we immediately go to Postgres to find out what that incident actually looks like right now.

RAG finds the candidate. The live store confirms the truth.

---

Why This Is a Category Error, Not Just a Bug

The instinct to use vector index metadata for status checks is understandable. The metadata is right there in the search result. Fetching from Postgres is an extra query. It feels redundant.

But the vector index and the live store are doing completely different jobs:

	pgvector index	Postgres live store
Purpose	Find semantically similar things	Track current state of things
Updated	When you explicitly re-index	On every state transition
Query type	”What’s similar to this?"	"What is this right now?”
Consistency	Eventual (when you index)	Immediate
Right tool for	Finding candidates	Confirming current truth

Using vector metadata to answer “what is the current status of this incident?” is like checking last year’s org chart to find out who’s managing a team today. The chart was accurate when it was printed. It might still be accurate. But it has no mechanism to stay current, so you can’t trust it for time-sensitive decisions.

The org chart is useful for finding names. HR’s live database is the source of truth for current roles. Those are different tools for different questions.

---

The General Rule

Any time you have a search index sitting in front of a live data store, the same principle applies:

Use the index to find candidates. Use the live store to confirm their current state.

This is true for pgvector. It’s also true for ChromaDB, Elasticsearch, Pinecone, Weaviate, or any other vector or search index you put in front of a database. Indexes are optimized for retrieval, not for reflecting real-time state. The moment you use index metadata to make a decision that depends on current state, you’ve introduced a staleness bug that will surface under exactly the wrong conditions — when an incident is active, when state is changing, when you most need the answer to be right.

The extra round-trip to the live store is not overhead. It’s correctness.

The fix wasn’t clever. It was just using each tool for what it’s designed for.