What Is Question Answering with Document Retrieval?

What Is Question Answering with Document Retrieval?

Question answering with document retrieval is the LLM workflow that combines a retriever with a generator to answer questions over a corpus. The user’s question is embedded, the retriever (dense vector search, BM25, hybrid, or a reranker stack) returns the top-K most relevant chunks, the chunks are concatenated into the prompt as context, and the LLM produces an answer grounded in that context. It is the canonical retrieval-augmented-generation (RAG) deployment for knowledge-intensive QA, replacing fine-tuning for most enterprise knowledge-base, support-bot, and developer-docs use cases. In a FutureAGI trace, the system appears as a retriever span followed by a generator span sharing a chunks payload.

Why It Matters in Production LLM and Agent Systems

The reason QA-with-retrieval dominates is that closed-book LLMs hallucinate on anything outside their parametric memory and can never see your private data. Retrieval gives you up-to-date, source-attributable answers and an audit trail. The cost is that the system has more failure modes than a closed-book LLM. A bad embedder retrieves irrelevant chunks; the generator invents a plausible answer anyway. A correct retriever pulls the right chunks but the chunks are too long and the model loses them in the middle. The reranker boosts a confident-but-wrong chunk over the correct one. The generator ignores the retrieved context entirely and answers from priors.

The pain shows up in user complaints and silent quality drift. A support team sees the bot give the right answer for “what is the refund policy?” yesterday and a wrong answer today because someone added a draft document to the knowledge base. An ML engineer cannot tell whether a regression came from the retriever, the reranker, or the prompt — the end-to-end answer-relevancy score dropped 4% but doesn’t say where.

In 2026, with agentic-RAG patterns adding query rewriting, multi-hop retrieval, and self-RAG critiquing inside the loop, single-score evaluation is no longer enough. You need step-level evaluators wired to every span.

How FutureAGI Handles QA with Document Retrieval

FutureAGI’s approach is layered evaluation at every stage of the retrieval-and-generation chain. At the retrieval layer, ContextRelevance scores how relevant the retrieved chunks are to the question, ContextPrecision scores ranking quality, and ContextRecall scores retrieval completeness against ground truth. At the generation layer, Faithfulness and Groundedness score whether the answer is supported by the retrieved chunks, ChunkAttribution scores which specific chunks the answer cites, and ChunkUtilization scores how much of the retrieved context the model actually used. At the user layer, AnswerRelevancy scores whether the response actually answers the question.

Concretely: a documentation team running a QA bot on traceAI-langchain with a Pinecone retriever ships every trace into FutureAGI. They sample 5% into an evaluation cohort and run ContextRelevance, Faithfulness, and AnswerRelevancy on each. When eval-fail-rate-by-cohort spikes, the layered scores point to the retriever — ContextRelevance dropped from 0.82 to 0.61 after a chunking-strategy change, while Faithfulness held steady. Without layered evaluation, the team would only have seen “answer quality fell” with no idea where to look. FutureAGI’s role is making each link in the chain individually scorable.

How to Measure or Detect It

QA-with-retrieval needs separate signals at retrieval, grounding, and answer layers:

• ContextRelevance: returns 0–1 for whether retrieved chunks are relevant to the query — the retrieval-layer alarm.
• ContextPrecision and ContextRecall: measure ranking and completeness if you have ground-truth chunk labels.
• Faithfulness: scores whether the answer is grounded in retrieved chunks — the hallucination-in-RAG alarm.
• ChunkAttribution and ChunkUtilization: surface chunks the model cited and how much of context it used.
• AnswerRelevancy: end-to-end quality of the answer relative to the question.
• eval-fail-rate-by-cohort sliced by retriever, embedder, prompt version: the canonical regression dashboard.

from fi.evals import ContextRelevance, Faithfulness, AnswerRelevancy

ctx = ContextRelevance()
faith = Faithfulness()
ans = AnswerRelevancy()

result = faith.evaluate(
    input="What is the refund policy?",
    output=generated_answer,
    context=retrieved_chunks,
)
print(result.score, result.reason)

Common Mistakes

• Scoring only the final answer. A 0.7 end-to-end score hides whether retrieval, grounding, or generation is the regression source.
• Skipping ContextRelevance because retrieval “works”. Retrievers drift silently with corpus updates; score them every release.
• Using the same model as judge and generator. Self-evaluation inflates scores; pin the judge to a different family.
• Long-context complacency. A 200K-window model can still ignore retrieved chunks in the middle — measure ChunkUtilization.
• One golden dataset forever. Production traffic drifts; sample real traces into your eval cohort continuously.