Your LLM Eval Failed. Which Input Broke It? Field-Level Eval Attribution in 2026

Originally published May 29, 2026.

Your eval dashboard lights up red. Faithfulness on the support-bot answer scored 0.41. Fail. Now what? The number tells you the answer was not grounded. It does not tell you whether the retriever pulled an irrelevant chunk, the user’s question was ambiguous, or the model ignored a perfectly good context. Three different bugs, three different teams, one useless number. So you do what everyone does: open the transcript and start reading.

That re-reading is the tax on every eval that returns a score and nothing else. This post is about removing it. We will define field-level eval attribution, show why a pass/fail score was never a debugging tool, and walk through turning a red eval into a one-line diagnosis with code you can run today.

What Is Field-Level Eval Attribution?

Field-level eval attribution is the practice of mapping a failed evaluation back to the specific input that caused it, instead of reporting a single pass/fail score. When a faithfulness or correctness eval fails, attribution names whether the retrieved context, the user question, the system prompt, or the model output is responsible. It turns an eval from a verdict into a diagnosis, so a developer fixes the actual broken input rather than re-reading the whole transcript.

The distinction matters because an eval reads several inputs and emits one number. A RAG correctness check looks at the question, the retrieved context, and the answer. When it fails, the root cause lives in one of those three, and the score flattens all three into a single digit. Attribution un-flattens it and names the field.

Why Isn’t a Pass/Fail Score a Debugging Tool?

A score answers “did it pass?” Debugging asks “why did it fail, and what do I change?” Those are different questions, and most eval stacks only answer the first.

Consider a single failing correctness eval on a RAG answer. The score is 0.4. Here are the candidate root causes, each with a different owner and a different fix:

• The context was wrong. The retriever returned chunks that do not contain the answer. Fix: the retriever, the embeddings, or the chunking. Owner: the RAG team.
• The question was ambiguous. The user asked something underspecified, so the retrieval and the answer both went sideways. Fix: query rewriting or clarification. Owner: the product flow.
• The output ignored the context. The context had the answer and the model drifted anyway. Fix: the prompt or the model. Owner: the prompt author.

A bare score cannot tell these apart, so the debugging step is a human reading transcripts and forming a hypothesis. At ten failures that is annoying. At ten thousand it is the reason your eval suite produces dashboards nobody acts on. The score found the failure; it did not localize it.

How Does Field-Level Attribution Work?

Attribution sits on top of an LLM-as-a-judge evaluation, not instead of it. The judge already produces two things: a score and a reason for the score. Attribution adds a third step that takes the reason and assigns it to a structured input key.

The mechanism is a second structured pass, not magic. The evaluator already holds the full set of inputs it scored (context, query, output, and any others the template defines) plus the judge’s reason for the low score. The localization pass takes that reason and weighs each input’s contribution to the failure, then returns the input key that is responsible as a structured label, rather than leaving that inference to a human reading the transcript.

The output is machine-readable: a field name, not a paragraph. That is what makes it routable. You can send context-attributed failures to one queue and output-attributed failures to another, automatically.

This is the difference between explainability and attribution. A judge reason is prose a human reads. An attribution is a label a pipeline acts on. It is the same instinct behind layering deterministic and LLM-judge evals: make each failure cheap to act on, not just cheap to detect.

How Do You Turn On Error Localization in Future AGI?

In Future AGI’s ai-evaluation SDK, field-level attribution is a single flag. You run the same evaluator you already use, with error_localizer=True, and the result carries an attribution payload next to the score.

from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
)

result = evaluator.evaluate(
    eval_templates=["groundedness"],
    inputs=[{
        "query": user_question,
        "context": retrieved_context,
        "output": model_answer,
    }],
    error_localizer=True,          # turn a score into a diagnosis
)

eval_result = result.eval_results[0]
print(eval_result.output)          # the score: e.g. 0.41 (fail)
print(eval_result.reason)          # the judge's reasoning
print(eval_result.error_localizer) # the responsible input field

The first two lines are a normal eval: a score and the judge’s reasoning. The third line is the part that ends the transcript-reading. Because the evaluator scores against 50+ built-in templates without needing ground-truth labels, you get attribution on open-ended production traffic, not just on a labeled test set.

The production pattern: score everything cheaply, then re-run only the failures with error_localizer=True. You pay the extra localization call only on red results, and every red result arrives pre-triaged with the field to fix.

How Does a Score-Only Eval Compare to Field-Level Attribution?

The production trade-off: score-only is the right default for the high-volume gate, and attribution is what you attach to the failures the gate catches. Run both, layered, not one or the other.

What Does Field-Level Attribution Look Like in a RAG Failure?

Say a support bot answers “Your refund will arrive in 3 to 5 business days,” groundedness scores 0.4, and the eval fails. Score-only, you open the transcript. With attribution on, the result names the responsible field.

• If attribution points at context, the retriever never pulled the refund-policy chunk; the model answered from priors. Fix retrieval, not the prompt.
• If attribution points at output, the refund-policy chunk was right there in context and the model said 3 to 5 days anyway when the policy says 7 to 10. Fix the prompt or escalate the model. This output-versus-context split is exactly the one that makes LLM hallucination so hard to debug from a score alone.

Same score, opposite fixes. The attribution is what tells you which one you are looking at before you have read a single line of the transcript. Multiply that across a daily eval run and the difference is whether root-cause is minutes or an afternoon.

Where It Falls Short

• Attribution is a diagnosis, not a proof. It is a model-produced lead, a strong one, but confirm it on high-stakes failures before you act on it blindly.
• It costs an extra pass. Localization adds a call per item, so enable it on failures and on a sample, not on every green eval.
• It inherits the rubric’s quality. Localization explains why the rubric failed; a vague rubric yields a vague attribution. Calibrate the eval first.

Why Should Attribution Be Part of Your Eval Stack?

Evals got good at finding failures and stayed bad at explaining them. A red dashboard with no attribution is a to-do list of transcript-reading. Field-level eval attribution closes that gap: it takes the judge’s reasoning and assigns the failure to the input that caused it, so the score points at the fix and the failure routes to the right owner. The score was always the easy half. The cause is the half that costs you the afternoon.

Want your next failed eval to name the input that broke it? Turn on error_localizer=True in Future AGI’s evaluation docs and stop reading transcripts to guess.