What Is Ground Truth Match?

What Is Ground Truth Match?

Ground truth match is an LLM-evaluation metric that checks whether a model output matches a known-correct label, answer, or field from a golden dataset. It belongs to reference-based evals and shows up in offline regression suites, production trace scoring, and classifier-style eval pipelines. FutureAGI exposes it through the GroundTruthMatch evaluator for the eval:GroundTruthMatch surface, where teams compare responses against gold references before approving prompt, model, retriever, or agent changes.

Why It Matters in Production LLM and Agent Systems

Ground truth match is the line between “the model sounded right” and “the model returned the expected answer.” If you ignore it on canonical-answer tasks, you ship label drift, extraction regressions, routing errors, and tool-call outputs that look plausible but fail downstream systems. A support classifier starts tagging refund requests as billing disputes. A SQL assistant returns the right-looking metric with the wrong region. A medical intake agent maps “chest tightness” to a low-priority queue because the label text was close, not equal.

The pain lands on different teams at once. ML engineers lose their clean regression signal. Product managers see unexplained changes in completion rate. SREs get alerts from downstream services instead of the model layer. Compliance teams cannot prove that audited prompts still choose approved labels. In logs, the symptoms are high semantic similarity but low gold-label agreement, rising eval_fail_rate on one cohort, and repeated mismatches concentrated in a specific prompt version, retriever branch, or model route.

Agentic systems make this stricter. One wrong intermediate label can pick the wrong tool, fetch the wrong record, and give the final model context that makes a bad answer look grounded. Ground truth match catches the first wrong discrete decision before it becomes a multi-step failure.

How FutureAGI Handles Ground Truth Match

FutureAGI’s approach is to keep the gold reference attached to the unit being evaluated: a dataset row, trace, or regression case. The specific FAGI surface is eval:GroundTruthMatch, implemented through the GroundTruthMatch evaluator in the eval catalog. Engineers attach it to rows that include input, response, and a gold column such as expected_response or ground_truth_label. The metric reported is ground-truth pass rate, usually sliced by dataset version, prompt version, model, and task cohort.

Example: an insurance intake agent classifies user messages into claim_status, new_claim, billing, or handoff. The team stores 2,000 reviewed examples in a FutureAGI Dataset, adds GroundTruthMatch through Dataset.add_evaluation, and runs it in CI before any prompt or model change. Production traces come in through traceAI-langchain; failed traces with low confidence or user escalation are sampled back into the dataset after human review. If GroundTruthMatch drops from 0.982 to 0.951 on the handoff cohort, the release is blocked and the failing rows become a targeted regression eval.

Unlike Ragas answer correctness, which often uses an LLM judge to compare a free-form answer to a reference, GroundTruthMatch is for canonical outputs. It is stricter, cheaper to run at scale, and easier to explain in an audit: the model either matched the approved answer under the chosen comparison rule or it did not.

How to Measure or Detect It

Use ground truth match only when a gold reference exists. The core signals are:

• fi.evals.GroundTruthMatch — compares response with the gold reference and returns an evaluator result that can be aggregated into pass rate.
• Gold-label agreement by cohort — split by prompt version, model, route, locale, and dataset version; a small drop usually means a real discrete-task regression.
• Mismatch reason sampling — inspect failed rows for label taxonomy drift, ambiguous gold answers, or prompt instructions that changed output format.
• User-feedback proxy — thumbs-down rate, escalation rate, or corrected-label rate on the same cohort; use it to find missing gold cases.
• Trace field coverage — confirm production traces preserve input, llm.output, and the expected label before scoring live traffic.

from fi.evals import GroundTruthMatch

evaluator = GroundTruthMatch()
result = evaluator.evaluate(
    response="refund_request",
    expected_response="refund_request",
)
print(result.score, result.reason)

Common Mistakes

• Using it on open-ended answers. If multiple phrasings are correct, ground truth match undercounts quality; use FactualConsistency, AnswerRelevancy, or FuzzyMatch.
• Letting gold labels drift from product taxonomy. A stale expected_response makes a good model look wrong and hides real regressions.
• Mixing task types in one pass rate. Extraction, classification, and routing deserve separate thresholds because their error costs differ.
• Ignoring ambiguous gold rows. If two reviewers disagree, the evaluator will amplify annotation noise, not model quality.
• Treating semantic similarity as enough. A label that is “close” can still call the wrong tool or violate a policy workflow.