What Is a RAGET Simple Question Hallucination Attack?

What Is a RAGET Simple Question Hallucination Attack?

A RAGET simple-question hallucination attack is a structured test case in the RAGET evaluation taxonomy that probes a retrieval-augmented generation system with single-fact, lookup-style questions. The answer is supposed to be in one document chunk, retrievable in one hop, with no synthesis required. The attack succeeds — meaning the RAG system fails — when the system fabricates an answer instead of grounding cleanly. Failure paths include the retriever missing the relevant chunk, the reranker boosting a confident-but-wrong chunk above the correct one, or the generator ignoring retrieved context and answering from priors. It is the floor regression eval for RAG: pass it, and you can move on to harder shapes; fail it, and something fundamental is broken.

Why It Matters in Production LLM and Agent Systems

If a RAG system fails on simple-question lookup, every other eval signal is suspect. A confident wrong answer to “what is the price of the enterprise plan?” — when the answer is one paragraph in the pricing page — is the most basic failure a knowledge bot can have, and the most damaging. Customers do not give second chances. The bot has demonstrated it cannot be trusted on the easy questions, so users will not trust it on the hard ones either.

The pain falls on RAG team owners and product leads. A QA engineer notices the bot answered the same simple question correctly last week and wrong this week — the cause is usually a corpus update, an embedder swap, or a chunking-strategy change that disturbed retrieval order. A compliance lead in a regulated industry sees a single-fact hallucination on regulated content (drug interaction, contract clause, account fee) and has to file an incident report. A model-upgrade lead pushes a “better” base model and sees simple-question pass rate drop because the new model is more confident about ignoring retrieval.

In 2026, simple-question hallucination is more dangerous than complex-question hallucination because users implicitly trust simple lookups. Confidence on the easy questions is what builds the false confidence that makes complex-question failures worse downstream.

How FutureAGI Handles RAGET Simple-Question Attacks

FutureAGI’s approach is to make the simple-question test cohort a permanent regression gate on every change to the RAG pipeline. Teams synthesise simple-question pairs from their corpus with synthetic-data-generation, load them into a Dataset, and attach Faithfulness, Groundedness, HallucinationScore, ContextRelevance, and ChunkAttribution via Dataset.add_evaluation(). The regression run is part of the release pipeline; any drop on the simple-question cohort blocks deploy until investigated.

Concretely: a docs-bot team running on traceAI-llamaindex with a Pinecone index ships every change through the simple-question regression suite. After a chunking-strategy change from 512-token to 256-token chunks, simple-question Faithfulness drops from 0.94 to 0.79 because semantic boundaries got cut mid-paragraph and the right answer is now spread across chunks the retriever is not pulling together. The regression blocks the merge; the team rolls the chunking change back. In production, a post-guardrail runs Faithfulness on every response below a threshold and routes those answers to “I’m not confident — let me get a human” rather than serving a wrong answer. FutureAGI is the offline regression gate plus the runtime guardrail.

How to Measure or Detect It

Simple-question RAG hallucination is measured at the grounding and retrieval layers:

• Faithfulness: 0–1 score for whether the answer is supported by retrieved chunks — the canonical alarm.
• Groundedness: companion metric that scores whether the response sticks to retrieved context.
• HallucinationScore: composite signal that combines Faithfulness with reference-answer comparison.
• ContextRelevance and ContextRecall: catch the upstream cause when retrieval missed the right chunk.
• ChunkAttribution: surfaces which chunks the response claims vs. what was actually retrieved.
• Simple-question cohort eval-fail-rate: the regression gate sliced by question shape.

from fi.evals import Faithfulness, Groundedness, HallucinationScore

faith = Faithfulness()
ground = Groundedness()
hallu = HallucinationScore()

result = faith.evaluate(
    input="What is the price of the enterprise plan?",
    output=generated_answer,
    context=retrieved_chunks,
)
print(result.score, result.reason)

Common Mistakes

• Skipping the simple-question regression because “it should never fail”. That is exactly when it does — corpus updates and chunking changes break it silently.
• Trusting the retriever’s top-1 ranking. Confident-but-wrong chunks get promoted; verify with ContextRelevance.
• Letting the model answer when retrieval returns no relevant chunks. Prompt the model to refuse rather than fabricate.
• No post-guardrail on production traffic. Offline pass-rate has a tail; use Faithfulness runtime to catch the tail.
• Comparing simple-question scores across model families without re-tuning prompts. A new model may need different grounding instructions to match the prior baseline.