What Is an LLM Hallucination?

What Is an LLM Hallucination?

An LLM hallucination is a production failure mode where a language model emits content that is fluent and confident but factually unsupported. The model invents a function signature, fabricates a legal citation, or asserts a number that nothing in its training data or retrieved context backs up. Because the surface text reads like every other correct answer, hallucinations slip through human review and load-test sampling. They surface across RAG, agent reasoning, summarisation, and structured extraction, and they need dedicated evaluators — a single regression eval is not enough.

Why It Matters in Production LLM and Agent Systems

On 2026-02-08 a large fintech rolled out a “policy assistant” chatbot that paraphrased benefit documents for HR. Three weeks later, an employee filed a wrongful-denial claim because the bot had cited a 90-day appeal window that did not exist in any policy document. Postmortem: the retriever returned the right chunk, but the model added a sentence that interpolated a number from a different document type. No alert fired. No eval caught it.

That is the canonical hallucination failure. It hits everyone in the chain: the engineer who shipped the prompt, the SRE who has no signal in the trace dashboard, the support team handling the escalation, and the compliance lead who must explain to auditors how a regulated communication slipped out unverified.

In agentic systems the cost compounds. A planner hallucinates a tool name in step 1; the executor in step 2 calls a non-existent endpoint and retries; the recovery agent in step 3 invents a justification for the failed call. By step 5 the trajectory is fiction the model has fully committed to. FutureAGI’s 2026 trace data shows that roughly one in eight long agent runs contains at least one hallucinated reasoning step that was never flagged by single-turn evals — which is why step-level hallucination scoring is now table stakes, not nice-to-have.

How FutureAGI Handles Hallucinations

FutureAGI’s approach is to detect hallucinations at three layers and prevent them at one. Detection: DetectHallucination (cloud template, Pass/Fail with a reason) runs on every answer span where context is available; HallucinationScore (local metric, continuous) trends the issue over time across deploys; Groundedness and Faithfulness partner with both for RAG-grounded variants. Prevention: the Agent Command Center pre-guardrail (the ProtectFlash lightweight check) blocks user inputs designed to elicit fabricated content (e.g. “make up a citation if you have to”) before they ever hit the model.

Concretely: a RAG team instruments their LangChain pipeline with traceAI-langchain. Every retrieval span carries retrieval.documents and every answer span carries llm.output. DetectHallucination is wired to the answer span and writes Pass/Fail back as a span event. The dashboard plots hallucination-fail-rate by route. When a model swap pushes the rate from 3% to 11%, the team opens the FutureAGI evaluation explorer, clusters failing reasons, and sees that the new model is inventing dates whenever it sees a partial date in context. They roll back, file a regression eval, and add a post-guardrail that strips date assertions not present verbatim in the retrieved chunks.

Unlike Ragas faithfulness, which scores claim-by-claim only inside RAG, FutureAGI’s hallucination stack works on free-form generation, agent reasoning, and structured outputs.

How to Measure or Detect It

Signals to wire up:

• fi.evals.DetectHallucination — Pass/Fail per response with a reason. Inputs: output + context.
• fi.evals.HallucinationScore — continuous 0–1 score combining multiple sub-checks; use for trending.
• OTel attributes llm.output.value and retrieval.documents — both must be present on the span for the evaluator to score against.
• Dashboard signal: hallucination-fail-rate-by-cohort — split by route, model, and prompt version.
• User-feedback proxy: thumbs-down rate within 60 seconds of an answer — strongly correlates with hallucinated outputs.

from fi.evals import DetectHallucination

evaluator = DetectHallucination()

result = evaluator.evaluate(
    output="The refund window is 60 days.",
    context="Refunds may be requested within 30 days of purchase."
)
print(result.score, result.reason)

Common Mistakes

• Treating fluency as a correctness signal. Hallucinated answers are usually the most fluent ones in your dataset — the model is comfortable making things up.
• Running hallucination evals only at offline regression time. Production drift hits at deploy boundaries; you need live trace-level scoring too.
• Using the same model family as both generator and judge. Self-evaluation systematically under-reports hallucinations from that family.
• Confusing hallucination with retrieval failure. If the context never had the fact, that is a context-relevance problem; pair DetectHallucination with ContextRelevance.
• Setting a single global threshold. Tolerable hallucination rates differ by domain — medical vs. marketing copy are not the same gate.