LangGraph Agent Evaluation: A 2026 Deep Tutorial
Learn LangGraph evaluation by scoring state transitions: node input, node output, routing choices and checkpoint replay, traced with traceAI and gated in CI.
Table of Contents
Picture a LangGraph agent that scores 0.93 on TaskCompletion against a fifty-query regression set. Every final answer looks fine. A week later, cost per query has doubled, and the trace shows why: the router took the expensive branch on every query that mentioned a date, the retry loop fired three times before the model fabricated an answer, and a reducer overwrote a constraint the planner had set two nodes upstream.
The final message was right. The state machine was broken. LangGraph evaluation has to score how the graph reached its answer, because a LangGraph agent is a state machine with typed channels, conditional edges, retry loops, sub-graphs, parallel fan-in, and a checkpointer that saves state at every super-step. The unit of evaluation is the state transition, not the assistant message.
This tutorial covers the four transition-level rubrics, the traceAI spans that expose each node and routing decision, the checkpoint-replay loop that turns flaky failures into regression cases, and the Future AGI loop from failing trajectory to shipped fix. Score the transitions or you ship a state machine you cannot reason about.
TL;DR
- Score four properties of every state transition: node input, returned update, routing decision, and checkpoint replay.
- traceAI’s
LangChainInstrumentorturns each node and routing function into its own span. - With a checkpointer, LangGraph saves state at every super-step, so a failing run replays from the step before it broke.
Why message-level eval misses graph failures
LangChain chains are linear. LangGraph is not. The failure modes that survive a message-level pass live in the seams the chain framing does not have.
- Conditional edges. A router reads state and picks one of N branches. A passing answer over the wrong branch is still a defect.
- Retry loops. A node can loop back to itself. A graph that retries a tool five times produces a correct answer at five times the cost.
- Typed channels. State is a set of typed channels with reducers (
add_messages,operator.add, custom). A reducer that overwrites where it should append silently breaks downstream nodes. - Sub-graphs. A node can be a compiled graph. Evaluation has to traverse parent-child topology, not flatten it.
- Parallel fan-in. A graph can fan out to N nodes and reduce. A merge that collides on a key drops data.
- Checkpointed memory. A regression in checkpoint write logic does not surface until turn three.
The final message averages over all of this. It moves when the graph is catastrophically wrong and stays flat while the state machine quietly degrades. Background on the broader axis-blindness pattern: agent passes evals, fails production and the definitive guide to AI agent evaluation in 2026.
LangGraph trajectory evaluation with LangSmith and agentevals
The LangChain tooling is the native starting point for LangGraph trajectory evaluation. The LangSmith guide to evaluating a graph scores end-to-end runs against a dataset, intermediate steps, and single nodes run on their own. The open-source agentevals package adds graph trajectory evaluators that check which nodes a run visited, by strict match against a reference or with an LLM judge.
A trajectory match tells you the path changed. It does not say which node read a stale state slice, which update overwrote a constraint, or why the router took the expensive branch. The four transition rubrics below answer that on the traces you already collect, and the framework-agnostic view of node, path, and outcome levels is in three levels of state-graph agent evaluation.
State transitions are the unit
A state transition is a four-tuple: (state_before, node, state_after, edge_taken). Every LangGraph execution is an ordered sequence of these plus the checkpoint metadata that lets you replay any prefix. The eval set is a list of expected transition sequences, not a list of expected final answers. Same primitive, four properties:

- Node-input correctness. Did the node receive the state it expected, with the channel schema respected and no stale checkpoint shadowing.
- Node-output correctness. Did the node produce a state diff that respects the channel schema and the node’s contract.
- Edge-routing correctness. On conditional edges, did the router pick the branch the rubric considers correct for the state at decision time.
- Checkpoint-replay determinism. Does re-running from a saved checkpoint with the same inputs reproduce the same downstream trajectory.
Each layer catches a different failure class. Together they point at the layer where a regression starts.
| Layer | What it scores | Future AGI evaluator |
|---|---|---|
| Node-input | Channel schema respected, state slice expected | Custom eval node_input_contract |
| Node-output | State diff respects schema and node intent | Custom eval state_diff_correctness |
| Edge-routing | Router picked correct branch given state | Custom eval routing_accuracy |
| Checkpoint-replay | Same checkpoint plus inputs reproduce trajectory | Code eval on TrajectoryHashMatch hashes |
Layer this on top of the standard answer-level templates (TaskCompletion, Groundedness, AnswerRefusal, EvaluateFunctionCalling). The templates grade the final response, and the four transition rubrics grade how the graph got there.
What traceAI captures from a LangGraph run
A transition-level rubric needs a transition-level span. traceAI’s LangChainInstrumentor traces LangGraph through LangChain’s own callbacks (more in tracing LangGraph conditional edges and retried nodes). Every run records:
- Each node as a span tagged
gen_ai.agent.graph.node_nameandnode_id input.value(state received) andoutput.value(update returned), as JSONlanggraph_step,langgraph_path, andlanggraph_checkpoint_nsper node- Each routing function as a child span, with the chosen branch in
output.value thread_idon every span when the run has one (a checkpointer requires it), also asgen_ai.conversation.id
Sub-graphs nest: a sub-graph’s spans sit under the node that invoked it. traceAI is built on OpenTelemetry, so a different exporter sends the same spans to Jaeger or any OTel-compatible backend without re-instrumenting.
Wiring traceAI into a LangGraph project
The instrumentation is callback-based, so you make no code changes inside node functions and pass no tracer through the graph compile. Install traceAI-langchain and fi-instrumentation-otel, register a tracer provider, and instrument once at process start.
from fi_instrumentation import register
from fi_instrumentation.fi_types import (
ProjectType,
EvalTag,
EvalTagType,
EvalSpanKind,
EvalName,
ModelChoices,
)
from traceai_langchain import LangChainInstrumentor
tracer_provider = register(
project_name="support_agent",
project_type=ProjectType.EXPERIMENT,
project_version_name="v1.4.2",
eval_tags=[
EvalTag(
type=EvalTagType.OBSERVATION_SPAN,
value=EvalSpanKind.TOOL,
eval_name=EvalName.EVALUATE_FUNCTION_CALLING,
model=ModelChoices.TURING_LARGE,
mapping={"input": "input.value", "output": "output.value"},
),
EvalTag(
type=EvalTagType.OBSERVATION_SPAN,
value=EvalSpanKind.AGENT,
eval_name=EvalName.TASK_COMPLETION,
model=ModelChoices.TURING_LARGE,
mapping={"input": "input.value", "output": "output.value"},
),
],
)
LangChainInstrumentor().instrument(tracer_provider=tracer_provider)
Compile and run any LangGraph as usual. TASK_COMPLETION fires on AGENT spans, and the root gets that kind only when the graph name contains agent, so compile the graph with name="support_agent". EvalTags score spans server-side as they arrive (instrumenting your AI agent with traceAI).
EvalTags need ProjectType.EXPERIMENT, so this setup fits pre-release and CI runs. For live traffic, register a project with ProjectType.OBSERVE and no eval_tags, then attach the same evals as a continuous eval task in Observe, the project type Error Feed scans.
Per-node rubrics for input contracts and state updates
Per-node rubrics gate the unit transition. They are the upstream signal that catches the bug before the chain-level metric averages it away.
node_input_contract asks whether the node received the state slice it expected. A planner node expects messages, user_intent, and available_tools populated. If the upstream router fired before user_intent was set, the planner runs on a state that violates its contract, and downstream nodes inherit the bug. The rubric:
from fi.evals import evaluate
# node_input: the planner span's input.value; PLANNER_CONTRACT: one line per required field
node_input_contract = evaluate(
prompt="""You are evaluating whether a graph node received the state it expects.
Node name: {node_name}
Node input contract (one line per required field): {node_contract}
State slice the node received (JSON): {state_slice}
Score 1.0 if every required field is populated with a value of the expected type.
Score 0.5 if one optional field is missing.
Score 0.0 if a required field is missing, null, or of the wrong type.
List any contract violations you found.""",
engine="llm",
model="gpt-6-luna",
node_name="planner",
node_contract=PLANNER_CONTRACT,
state_slice=node_input,
)
state_diff_correctness asks whether the node’s output diff respects the channel schema and the node’s stated intent. traceAI records the returned update as JSON in output.value, which matters: an LLM judge over a free-text summary will miss an append that became a replace, dict.update vs dict.setdefault, and type coercions that only manifest two nodes later. The structured update makes the rubric grade what actually happened.
state_diff_correctness = evaluate(
prompt="""You are evaluating whether a graph node updated shared state correctly.
Node name: {node_name}
Node intent (one line): {node_intent}
Channel schema (JSON): {channel_schema}
State diff produced (JSON): {state_diff}
Score 1.0 if the diff matches the intent and preserves all invariants.
Score 0.0 if the node dropped a required field, corrupted a field type, used the wrong
reducer (e.g., replaced where it should have appended), or violated an invariant.
List any broken invariants you found.""",
engine="llm",
model="gpt-6-luna",
node_name="planner",
node_intent=PLANNER_INTENT,
channel_schema=CHANNEL_SCHEMA,
state_diff=node_output,
)
Practical scoping: do not score every node with an LLM judge. A large graph has many state mutations and scoring all of them gets expensive. Pick the nodes whose output is read by multiple downstream consumers (planners, routers, reducers after parallel fan-in), score those at every run, and sample the rest.
Track per-node state_diff_correctness as a regression metric. When a node’s score drops, the fix is often a reducer change or a clarifying sentence in the node prompt. Tool nodes get the EVALUATE_FUNCTION_CALLING eval wired through the EvalTag above, and the evaluating tool-calling agents in 2026 guide covers the rubric design in depth.
Edge-routing rubric for silent graph failures
Conditional edges are the hardest LangGraph defect class because the graph still answers when the wrong branch was taken. A drop in routing accuracy on one decision node shows up in cost and latency before the final-answer metric moves.
routing_accuracy = evaluate(
prompt="""You are evaluating an agent's routing decision in a directed graph.
State at the decision point (JSON): {state}
Branches available at this node: {available_branches}
Branch the agent chose: {chosen_branch}
Score 1.0 if the chosen branch is the correct one given the state.
Score 0.0 if a different branch would have been clearly more appropriate.
Score 0.5 if the choice is defensible but not optimal.
Explain your reasoning in one sentence.""",
engine="llm",
model="gpt-6-luna",
state=router_input,
available_branches="search, answer",
chosen_branch=router_output,
)
On the Evals list, save the rubric as a custom eval named routing_accuracy, with the branch list in its prompt and the variables {{state}} and {{chosen_branch}}, then attach it to the Experiment project with the EvalTag below on CHAIN spans. An EvalTag scores every CHAIN span, node spans included; Observe eval tasks can filter spans by attribute.
EvalTag(
type=EvalTagType.OBSERVATION_SPAN,
value=EvalSpanKind.CHAIN,
eval_name="routing_accuracy",
mapping={
"state": "input.value",
"chosen_branch": "output.value",
},
)
Track per-decision-node routing accuracy on the regression set as a first-class metric. The signal that points at a fix is per-node: a drifting router prompt is a one-prompt fix, while a small TaskCompletion drop means bisecting the graph. Patterns for building custom judge rubrics: LLM-as-judge best practices in 2026.
Checkpoint replay with the LangGraph checkpointer
The checkpointer is the durability primitive that lets a long-running graph pause and resume. It is also an evaluation lever. A checkpointer saves the full graph state at every super-step, so a failure becomes a regression case once you have its thread and checkpoint ID. Replay re-executes the later nodes, so pin temperature and stub external tools when you need the same trajectory twice.
The pattern:
- Configure the checkpointer (
PostgresSaverin production,SqliteSaverorInMemorySaverlocally) and passdurability="sync"or"async"when you invoke or stream the graph;"exit"saves only when the run exits. - On a failing run, take the
thread_idfrom the trace and pick the checkpoint whosenextis the failing node fromgraph.get_state_history(config). - Replay the graph from that checkpoint:
graph.invoke(None, config={"configurable": {"thread_id": "...", "checkpoint_id": "..."}}). - Score the downstream trajectory with the four-layer evaluator stack.
- A/B prompt changes by replaying the same checkpoint against two graph versions and scoring the diff. That is a controlled experiment.
A TrajectoryHashMatch deterministic check sits beside the LLM-judge layer: hash the ordered sequence of (node_name, state_diff) tuples and assert the hash is stable across replays. A drift means the graph is non-deterministic in a way it should not be (an unsignalled temperature > 0, a tool call that depends on wall-clock time, a parallel reduce that did not stabilise ordering).
The hash check costs nothing per replay and catches a class of bug that ages into a production flake. Without checkpoint replay you are guessing whether a bug reproduces. With it you are running an experiment.
Building a regression set from checkpoints
Synthetic test inputs reflect the test author’s assumptions. Sample real production checkpoints instead.
- Filter spans where an eval score dropped below threshold.
- Group by transition shape: cluster on the failing node’s
gen_ai.agent.graph.node_idand the branch its routing span returned. - Promote representative checkpoints from each cluster, paired with the expected downstream trajectory annotated by a reviewer.
- Tag with metadata: which release shipped the regression, which user segment, which rubric dropped.
- Re-run on every PR by replaying the saved checkpoints. CI gate fails the PR if any rubric falls below its baseline threshold.
Every incident becomes a replayable test case. For threshold-calibration cadence: LLM evaluation playbook for 2026.
Closing the loop with Error Feed and prompt optimization
Error Feed reads a sample of an Observe project’s traces, decides what went wrong in each, and groups traces that failed the same way into one issue. Each issue names a fix layer (Prompt, Tools, Orchestration, or Guardrails) and links the traces, spans, and sessions behind it. A root cause analysis returns a written root cause and a proposed fix.
Issue shapes to expect on a LangGraph project:
- “Retry loop on tool X.” Ambiguous tool description or a node prompt that does not acknowledge tool failure.
- “Wrong conditional edge on date-range queries.” A router prompt that does not handle a specific state shape.
- “Reducer drops constraint after parallel fan-in.” A reducer function bug, not a prompt.
- “Sub-graph silently returned partial state.” A sub-graph key the parent schema does not declare, which LangGraph drops.
- “Checkpoint replay diverges from production trajectory.” Hidden non-determinism (wall-clock, unsignalled temperature, unstable parallel ordering).
Once an issue points at a node or router prompt, tune that prompt with the agent-learning-kit prompt optimizers, scored on your evals: RandomSearchOptimizer, BayesianSearchOptimizer, MetaPromptOptimizer, ProTeGi, GEPAOptimizer, and PromptWizardOptimizer. Give each node or router prompt its own run, and EarlyStoppingConfig stops a run that stalls or hits a score threshold. Background: automated optimization for agent systems.
How Future AGI ships this for LangGraph
-
agent-learning-kit (Apache-2.0).
pip install agent-learning-kit;evaluate()runs local metrics such astask_completionandgroundedness, cloud Turing templates such asevaluate_function_calling, or your own LLM-as-judge rubric, in CI. -
traceAI (Apache-2.0).
LangChainInstrumentortraces LangGraph nodes, routing functions, tools and LLM calls, withEvalTagserver-side evals. -
Future AGI Platform. Custom evals, continuous eval tasks on Observe spans, and Error Feed with five error groups, twenty categories and four fix layers.
-
Optimizers in agent-learning-kit. Six prompt optimizers scored on your evals.
-
Agent Command Center. A self-hostable, OpenAI-compatible gateway for the LLM calls under every node, with routing, failover, caching, and guardrails. Future AGI is SOC 2 Type II certified.
If your LangGraph is a five-node linear pipeline with no conditional edges, a single TaskCompletion rubric is enough. The four-layer stack earns its weight on graphs with real branching, retries, sub-graphs, or long-running checkpointed state. Framework-choice background: CrewAI vs LangGraph vs AutoGen and the best LangGraph alternatives.
What to do this week
One graph, end to end. Five steps.
- Install
fi-instrumentation-otelandtraceAI-langchain. Callregister()andLangChainInstrumentor().instrument()in process bootstrap. Verify node spans carrythread_id. - Turn on the checkpointer if you have not. Use
durability="sync"or"async". The replay loop is only useful if the checkpoints exist. - Add EvalTags for
EVALUATE_FUNCTION_CALLING,TASK_COMPLETION, and therouting_accuracyandstate_diff_correctnesscustom evals. - Open Error Feed on the Observe project after a week of traffic, once its sampling rate is above 0. Promote the failing traces behind two issues into a regression set built from checkpoints, not synthetic inputs. Replay each checkpoint in CI on every PR.
- Run the agent-learning-kit optimizers on the regression set with
BayesianSearchOptimizerandEarlyStoppingConfig. Tune the lowest-scoring router prompt as a single study target. Verify the winner does not regress any of the four layers.
Back to the agent from the opening: a 0.93 TaskCompletion score hid a router on the expensive branch, a retry loop, and a reducer that dropped a constraint. Per-node rubrics catch the reducer, the routing rubric catches the branch, and a replayed checkpoint shows the fix holds before the next deploy.
Trace your LangGraph agent and score it in Future AGI
Related reading
- Evaluating AutoGen Agents (2026)
- Evaluating CrewAI Agents (2026)
- Trace and Debug Multi-Agent Systems
Sources
Frequently Asked Questions
What is the LangGraph evaluation framework used for?
How do you evaluate a LangGraph agent?
How do you test LangGraph agents?
What is LangGraph trajectory evaluation?
Which metrics should you track to evaluate a LangGraph agent?
LangGraph is an MIT library for stateful agent graphs. Learn what LangGraph is and how StateGraph, nodes, edges, reducers and checkpointers work in version 1.x.
2026 working pattern for AI agent evaluation: six dimensions, six rubrics, a 4-D trajectory score, and a CI gate that beats aggregate scoring.
Tool-calling eval is four problems stacked: tool selection, argument extraction, result utilization, error recovery. Most posts grade only the first.