Why Did My RAG Agent Get Worse? A Live Autopsy of a Degrading RAG Pipeline

Watch the Webinar

Why Did My RAG Agent Get Worse? The TL;DR

Most RAG agents ship well on day zero. The failure shows up weeks later, when the data has grown and the same questions start returning worse answers.

What This Webinar Covers

A RAG agent that passes its tests in a clean environment often loses accuracy in production. The knowledge base keeps growing, the same documents get ingested more than once, and retrieval slowly returns duplicates instead of the passage the agent needs. The agent code has not changed, but the answers have.

This session runs a live autopsy on exactly that problem. Rishav Hada from Future AGI and Dylan Couzon from Qdrant take a working Pokédex RAG agent built on LangChain, trace it end to end, and watch its accuracy fall as the dataset expands from one Pokémon generation to eight.

Then they read the evaluation metrics to locate each fault and repair it one change at a time.

The goal is a repeatable method, not a single trick. You see how to tell a retrieval problem apart from a generation problem, and how to prove that a fix helped before you ship it.

Who Should Watch

AI and ML engineers, MLOps practitioners, and teams shipping retrieval augmented generation in production whose RAG pipeline passed its tests but started regressing once real data and real users arrived.

Why You Should Watch: From Duplicate Chunks to Re-Ranking and Metadata Filters

• How a Pokédex RAG agent on LangChain and Qdrant loses accuracy as its data grows, dropping from about 67% to 39% on the raw pipeline.
• Reading chunk attribution, chunk utilization, context relevance, groundedness, and answer correctness in Future AGI to find the failing step.
• Deduplicating a Qdrant collection and the jump in answer correctness that follows.
• Upgrading the embedding model from MiniLM to BGE and what happens to recall.
• Adding hybrid search and a ColBERT re-ranker, and why a single fix can push some metrics down.
• Using metadata filters so the agent serves only current data.

Key Insight

Retrieval quality and generator behavior are separate failure modes. You cannot fix what you cannot see, so trace the agent first, then read the metrics against a golden set before and after each change. The numbers and the traces together tell you what actually broke.

The Four Fixes That Took the RAG Agent From ~52% to 92%

1. Deduplicate the Collection

Re-ingesting the same documents left the collection full of duplicates, with one entry hitting an 80% duplicate rate. The agent reads only the top five results, so when copies fill those slots, the correct passage sits at position seven and never gets used.

Collecting the duplicate IDs and removing about 16,000 of them with the Qdrant delete API lifted answer correctness from 0.57 to 0.76. Clean data is the foundation, and chunking strategy decides how much of it survives retrieval.

2. Upgrade the Embedding Model

A small MiniLM model with 384 dimensions is fast and cheap to store, but it misses fine distinctions and struggles with negation, so it confused Pichu with Pikachu. Switching to BGE with 1024 dimensions took recall at five from 0.64 to a perfect 1.0 and raised answer correctness to 92%.

The trade is higher latency and more storage, which is why embedding model choice is a decision you measure against your own data.

3. Add Hybrid Search and Re-Ranking

Hybrid search combines dense embeddings, which capture meaning, with sparse embeddings, which match exact terms. Fusing both and then re-ranking the shortlist with a ColBERT late interaction model pushed context relevance to an all time high of 99%. Answer correctness, though, fell to 86% and groundedness to 84%.

With many relevant chunks in context, the generator started confusing near identical candidates, picking Sandshrew when Sandslash was the correct answer. This is where a re-ranker for RAG earns its keep only if you measure the generation side too.

4. Filter on Metadata

Some answers depend on which version of the data is current. A boolean payload index in Qdrant lets the agent filter on an is_current flag and serve only the latest generation, which finally fixed a stubborn question about Steel type resistances that older data kept answering wrong.

The same filter pattern extends to brand, category, price, or location for any production use case.

How Future AGI Evaluates and Observes a RAG Pipeline

Future AGI is the evaluation and observability layer for the RAG agent, whatever stack it runs on. Tracing is a few lines with the LangChain instrumentor, after which every run appears as a trace with its full span tree: the LLM call, each Pokédex search, and the retrieved context.

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from traceai_langchain import LangChainInstrumentor

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="rag_agent",
)
LangChainInstrumentor().instrument(tracer_provider=trace_provider)

With traces flowing, you attach evaluations to the spans. The session sets up chunk utilization and chunk attribution to measure how much retrieved context the generator actually uses, context relevance to check the retrieval itself, and groundedness plus hallucination to check the final answer.

Answer correctness runs against a golden set of expected answers. You filter by span kind and by whether retrieved context is present, map the variables, and run the evaluations on live traffic or on historical traces, with a sampling rate when volume is high.

Reading those metrics is the skill the webinar keeps returning to. A high context relevance with a falling answer correctness points at the generator, not the retriever. That separation is the core of RAG observability, and it is why a pipeline can pass every eval and still fail in production.

When there is no golden answer, an error feed surfaces the likely problems straight from the traces so you know where to look next.

Watch the Webinar and Explore Future AGI

The full session is gated above. For deeper coverage of the topics it touches, see:

• Advanced chunking techniques for RAG
• What is RAG observability?
• Best RAG debugging tools for 2026
• Best rerankers for RAG
• When your agent passes evals but fails in production

Qdrant is the open-source vector search engine used for the demo. See qdrant.tech for the vector database and docs.futureagi.com for tracing and evaluation.

Visit Future AGI | Quickstart docs | Book a demo