Context Failure Modes in My RAG Pipeline

The four failure modes, in one line each

Plus context rot: as a long-running agent accumulates turns and tool outputs, the model gets worse at using what is in the window. This happens before the token limit, so having room left does not prevent it.

Mapping each failure to a stage

1 · Poisoning: check ingestion

Risk: Chroma holds outdated, duplicate or badly extracted chunks, and nothing stops them from being retrieved.

• Version metadata on every chunk (version, effective_date, status).
• Freshness validation at ingest and active/inactive status per source.
• Provenance on every retrieved chunk (source, page, ingest run).
• Filter on metadata at query time, not just similarity: filter={"status": "active"}. Syntax varies by vector DB.

2 · Clash: check retrieval and reranking

Risk: the reranker puts two highly relevant chunks on top that contradict each other.

• Carry version and source through reranking. Don't strip metadata.
• Define source authority (registry > current policy > FAQ > archived).
• Detect conflicting facts and resolve them before generation. Never leave version management to the LLM.

3 · Confusion: check LangGraph orchestration

Risk: nodes or tools with overlapping responsibilities.

• One clear responsibility per node.
• Tool docstrings that state side effects and when not to use the tool ("this changes order status; it does NOT issue a refund").
• Explicit conditional routing: intent → validate → route (refund / cancel / clarify) → human approval for money-moving actions.
• Structured outputs between nodes, not free text.

4 · Distraction: check context assembly

Risk: too many chunks, verbose tool outputs and stale history all reach the prompt.

• Retrieval limits and a relevance threshold after reranking.
• Trim tool output to what the next decision needs (title, snippet, url; top 3).
• Route unrelated requests to a separate workflow instead of mixing them into the active task.
• Compact task state in the prompt; full raw records kept separately for audit.

The pipeline I would build

The architectural change

Before: retrieval output goes straight into the prompt.

context = "\n\n".join(doc.page_content for doc in retrieved_docs)
response = llm.invoke(prompt.format(context=context))

After: a validation and assembly stage sits between retrieval and generation.

def build_context(documents, k=5):
    selected = []
    for doc in documents:
        m = doc.metadata
        if m.get("status") != "active":
            continue
        selected.append({"source": m.get("source"),
                         "version": m.get("version"),
                         "content": doc.page_content})
    return selected[:k]

Production version adds date applicability, source authority, duplicate handling and conflict detection. Retrieved text is not trusted context until it passes this stage.

Audit checklist for my pipeline

1. Inspect Chroma metadata for all 11 sources.
2. Check whether versions and timestamps are recorded.
3. Find whether outdated or duplicate chunks can reach results.
4. Compare top 5 chunks before and after reranking.
5. Check whether contradictory chunks are detected.
6. Inspect the actual prompt sent to GPT-4o-mini.
7. Measure tokens entering the generation step.
8. Inspect LangGraph state for stale tool outputs.
9. Write four test cases: poisoning, clash, confusion, distraction.
10. Run the eval harness before and after each fix.

Retrieval, reranking and prompt assembly are three different stages, and each one can introduce a failure. Look at what the model actually receives, not just what Chroma returns.

Interview answers

Q: Your retriever returns an outdated doc, the reranker scores it highest, and the model answers wrong. What failed?
Context poisoning. The model reasoned correctly from bad input. The fix goes upstream: version metadata, active-status filtering and source validation. Relevance is not authority.

Q: Two nodes return different answers about the same policy, with no version info. What is it?
Context clash. I resolve it at the data layer with versioning and an authority order, not by hoping the model guesses the newer one.

Q: The agent passes 30 raw tool responses and 20 irrelevant chunks to the LLM. What do you do?
Context filtering and compression: relevance threshold, trimmed tool outputs, a compact state object, and raw history moved to audit logs. Raising temperature or adding more system instructions does not fix this.