Guides

LangGraph Agent Evaluation: A 2026 Deep Tutorial

Learn LangGraph evaluation by scoring state transitions: node input, node output, routing choices and checkpoint replay, traced with traceAI and gated in CI.

· Updated
· 11 min read
langgraph agent-evaluation llm-evaluation traceai agent-observability 2026
Banner with the title LangGraph Agent Evaluation: A 2026 Deep Tutorial beside a line diagram linking a state transition hub to node input, node output, edge routing and checkpoint replay.
Table of Contents

Picture a LangGraph agent that scores 0.93 on TaskCompletion against a fifty-query regression set. Every final answer looks fine. A week later, cost per query has doubled, and the trace shows why: the router took the expensive branch on every query that mentioned a date, the retry loop fired three times before the model fabricated an answer, and a reducer overwrote a constraint the planner had set two nodes upstream.

The final message was right. The state machine was broken. LangGraph evaluation has to score how the graph reached its answer, because a LangGraph agent is a state machine with typed channels, conditional edges, retry loops, sub-graphs, parallel fan-in, and a checkpointer that saves state at every super-step. The unit of evaluation is the state transition, not the assistant message.

This tutorial covers the four transition-level rubrics, the traceAI spans that expose each node and routing decision, the checkpoint-replay loop that turns flaky failures into regression cases, and the Future AGI loop from failing trajectory to shipped fix. Score the transitions or you ship a state machine you cannot reason about.

TL;DR

  • Score four properties of every state transition: node input, returned update, routing decision, and checkpoint replay.
  • traceAI’s LangChainInstrumentor turns each node and routing function into its own span.
  • With a checkpointer, LangGraph saves state at every super-step, so a failing run replays from the step before it broke.

Why message-level eval misses graph failures

LangChain chains are linear. LangGraph is not. The failure modes that survive a message-level pass live in the seams the chain framing does not have.

  • Conditional edges. A router reads state and picks one of N branches. A passing answer over the wrong branch is still a defect.
  • Retry loops. A node can loop back to itself. A graph that retries a tool five times produces a correct answer at five times the cost.
  • Typed channels. State is a set of typed channels with reducers (add_messages, operator.add, custom). A reducer that overwrites where it should append silently breaks downstream nodes.
  • Sub-graphs. A node can be a compiled graph. Evaluation has to traverse parent-child topology, not flatten it.
  • Parallel fan-in. A graph can fan out to N nodes and reduce. A merge that collides on a key drops data.
  • Checkpointed memory. A regression in checkpoint write logic does not surface until turn three.

The final message averages over all of this. It moves when the graph is catastrophically wrong and stays flat while the state machine quietly degrades. Background on the broader axis-blindness pattern: agent passes evals, fails production and the definitive guide to AI agent evaluation in 2026.

LangGraph trajectory evaluation with LangSmith and agentevals

The LangChain tooling is the native starting point for LangGraph trajectory evaluation. The LangSmith guide to evaluating a graph scores end-to-end runs against a dataset, intermediate steps, and single nodes run on their own. The open-source agentevals package adds graph trajectory evaluators that check which nodes a run visited, by strict match against a reference or with an LLM judge.

A trajectory match tells you the path changed. It does not say which node read a stale state slice, which update overwrote a constraint, or why the router took the expensive branch. The four transition rubrics below answer that on the traces you already collect, and the framework-agnostic view of node, path, and outcome levels is in three levels of state-graph agent evaluation.

State transitions are the unit

A state transition is a four-tuple: (state_before, node, state_after, edge_taken). Every LangGraph execution is an ordered sequence of these plus the checkpoint metadata that lets you replay any prefix. The eval set is a list of expected transition sequences, not a list of expected final answers. Same primitive, four properties:

One LangGraph state transition from state_before through the node and state_after to the routing edge, with the rubrics that score the input, the state diff and the routing choice, the checkpoint save and replay loop, and a wrong branch that still answers

  1. Node-input correctness. Did the node receive the state it expected, with the channel schema respected and no stale checkpoint shadowing.
  2. Node-output correctness. Did the node produce a state diff that respects the channel schema and the node’s contract.
  3. Edge-routing correctness. On conditional edges, did the router pick the branch the rubric considers correct for the state at decision time.
  4. Checkpoint-replay determinism. Does re-running from a saved checkpoint with the same inputs reproduce the same downstream trajectory.

Each layer catches a different failure class. Together they point at the layer where a regression starts.

LayerWhat it scoresFuture AGI evaluator
Node-inputChannel schema respected, state slice expectedCustom eval node_input_contract
Node-outputState diff respects schema and node intentCustom eval state_diff_correctness
Edge-routingRouter picked correct branch given stateCustom eval routing_accuracy
Checkpoint-replaySame checkpoint plus inputs reproduce trajectoryCode eval on TrajectoryHashMatch hashes

Layer this on top of the standard answer-level templates (TaskCompletion, Groundedness, AnswerRefusal, EvaluateFunctionCalling). The templates grade the final response, and the four transition rubrics grade how the graph got there.

What traceAI captures from a LangGraph run

A transition-level rubric needs a transition-level span. traceAI’s LangChainInstrumentor traces LangGraph through LangChain’s own callbacks (more in tracing LangGraph conditional edges and retried nodes). Every run records:

  • Each node as a span tagged gen_ai.agent.graph.node_name and node_id
  • input.value (state received) and output.value (update returned), as JSON
  • langgraph_step, langgraph_path, and langgraph_checkpoint_ns per node
  • Each routing function as a child span, with the chosen branch in output.value
  • thread_id on every span when the run has one (a checkpointer requires it), also as gen_ai.conversation.id

Sub-graphs nest: a sub-graph’s spans sit under the node that invoked it. traceAI is built on OpenTelemetry, so a different exporter sends the same spans to Jaeger or any OTel-compatible backend without re-instrumenting.

Wiring traceAI into a LangGraph project

The instrumentation is callback-based, so you make no code changes inside node functions and pass no tracer through the graph compile. Install traceAI-langchain and fi-instrumentation-otel, register a tracer provider, and instrument once at process start.

from fi_instrumentation import register
from fi_instrumentation.fi_types import (
    ProjectType,
    EvalTag,
    EvalTagType,
    EvalSpanKind,
    EvalName,
    ModelChoices,
)
from traceai_langchain import LangChainInstrumentor

tracer_provider = register(
    project_name="support_agent",
    project_type=ProjectType.EXPERIMENT,
    project_version_name="v1.4.2",
    eval_tags=[
        EvalTag(
            type=EvalTagType.OBSERVATION_SPAN,
            value=EvalSpanKind.TOOL,
            eval_name=EvalName.EVALUATE_FUNCTION_CALLING,
            model=ModelChoices.TURING_LARGE,
            mapping={"input": "input.value", "output": "output.value"},
        ),
        EvalTag(
            type=EvalTagType.OBSERVATION_SPAN,
            value=EvalSpanKind.AGENT,
            eval_name=EvalName.TASK_COMPLETION,
            model=ModelChoices.TURING_LARGE,
            mapping={"input": "input.value", "output": "output.value"},
        ),
    ],
)

LangChainInstrumentor().instrument(tracer_provider=tracer_provider)

Compile and run any LangGraph as usual. TASK_COMPLETION fires on AGENT spans, and the root gets that kind only when the graph name contains agent, so compile the graph with name="support_agent". EvalTags score spans server-side as they arrive (instrumenting your AI agent with traceAI).

EvalTags need ProjectType.EXPERIMENT, so this setup fits pre-release and CI runs. For live traffic, register a project with ProjectType.OBSERVE and no eval_tags, then attach the same evals as a continuous eval task in Observe, the project type Error Feed scans.

Per-node rubrics for input contracts and state updates

Per-node rubrics gate the unit transition. They are the upstream signal that catches the bug before the chain-level metric averages it away.

node_input_contract asks whether the node received the state slice it expected. A planner node expects messages, user_intent, and available_tools populated. If the upstream router fired before user_intent was set, the planner runs on a state that violates its contract, and downstream nodes inherit the bug. The rubric:

from fi.evals import evaluate

# node_input: the planner span's input.value; PLANNER_CONTRACT: one line per required field
node_input_contract = evaluate(
    prompt="""You are evaluating whether a graph node received the state it expects.

Node name: {node_name}
Node input contract (one line per required field): {node_contract}
State slice the node received (JSON): {state_slice}

Score 1.0 if every required field is populated with a value of the expected type.
Score 0.5 if one optional field is missing.
Score 0.0 if a required field is missing, null, or of the wrong type.

List any contract violations you found.""",
    engine="llm",
    model="gpt-6-luna",
    node_name="planner",
    node_contract=PLANNER_CONTRACT,
    state_slice=node_input,
)

state_diff_correctness asks whether the node’s output diff respects the channel schema and the node’s stated intent. traceAI records the returned update as JSON in output.value, which matters: an LLM judge over a free-text summary will miss an append that became a replace, dict.update vs dict.setdefault, and type coercions that only manifest two nodes later. The structured update makes the rubric grade what actually happened.

state_diff_correctness = evaluate(
    prompt="""You are evaluating whether a graph node updated shared state correctly.

Node name: {node_name}
Node intent (one line): {node_intent}
Channel schema (JSON): {channel_schema}
State diff produced (JSON): {state_diff}

Score 1.0 if the diff matches the intent and preserves all invariants.
Score 0.0 if the node dropped a required field, corrupted a field type, used the wrong
reducer (e.g., replaced where it should have appended), or violated an invariant.

List any broken invariants you found.""",
    engine="llm",
    model="gpt-6-luna",
    node_name="planner",
    node_intent=PLANNER_INTENT,
    channel_schema=CHANNEL_SCHEMA,
    state_diff=node_output,
)

Practical scoping: do not score every node with an LLM judge. A large graph has many state mutations and scoring all of them gets expensive. Pick the nodes whose output is read by multiple downstream consumers (planners, routers, reducers after parallel fan-in), score those at every run, and sample the rest.

Track per-node state_diff_correctness as a regression metric. When a node’s score drops, the fix is often a reducer change or a clarifying sentence in the node prompt. Tool nodes get the EVALUATE_FUNCTION_CALLING eval wired through the EvalTag above, and the evaluating tool-calling agents in 2026 guide covers the rubric design in depth.

Edge-routing rubric for silent graph failures

Conditional edges are the hardest LangGraph defect class because the graph still answers when the wrong branch was taken. A drop in routing accuracy on one decision node shows up in cost and latency before the final-answer metric moves.

routing_accuracy = evaluate(
    prompt="""You are evaluating an agent's routing decision in a directed graph.

State at the decision point (JSON): {state}
Branches available at this node: {available_branches}
Branch the agent chose: {chosen_branch}

Score 1.0 if the chosen branch is the correct one given the state.
Score 0.0 if a different branch would have been clearly more appropriate.
Score 0.5 if the choice is defensible but not optimal.

Explain your reasoning in one sentence.""",
    engine="llm",
    model="gpt-6-luna",
    state=router_input,
    available_branches="search, answer",
    chosen_branch=router_output,
)

On the Evals list, save the rubric as a custom eval named routing_accuracy, with the branch list in its prompt and the variables {{state}} and {{chosen_branch}}, then attach it to the Experiment project with the EvalTag below on CHAIN spans. An EvalTag scores every CHAIN span, node spans included; Observe eval tasks can filter spans by attribute.

EvalTag(
    type=EvalTagType.OBSERVATION_SPAN,
    value=EvalSpanKind.CHAIN,
    eval_name="routing_accuracy",
    mapping={
        "state": "input.value",
        "chosen_branch": "output.value",
    },
)

Track per-decision-node routing accuracy on the regression set as a first-class metric. The signal that points at a fix is per-node: a drifting router prompt is a one-prompt fix, while a small TaskCompletion drop means bisecting the graph. Patterns for building custom judge rubrics: LLM-as-judge best practices in 2026.

Checkpoint replay with the LangGraph checkpointer

The checkpointer is the durability primitive that lets a long-running graph pause and resume. It is also an evaluation lever. A checkpointer saves the full graph state at every super-step, so a failure becomes a regression case once you have its thread and checkpoint ID. Replay re-executes the later nodes, so pin temperature and stub external tools when you need the same trajectory twice.

The pattern:

  1. Configure the checkpointer (PostgresSaver in production, SqliteSaver or InMemorySaver locally) and pass durability="sync" or "async" when you invoke or stream the graph; "exit" saves only when the run exits.
  2. On a failing run, take the thread_id from the trace and pick the checkpoint whose next is the failing node from graph.get_state_history(config).
  3. Replay the graph from that checkpoint: graph.invoke(None, config={"configurable": {"thread_id": "...", "checkpoint_id": "..."}}).
  4. Score the downstream trajectory with the four-layer evaluator stack.
  5. A/B prompt changes by replaying the same checkpoint against two graph versions and scoring the diff. That is a controlled experiment.

A TrajectoryHashMatch deterministic check sits beside the LLM-judge layer: hash the ordered sequence of (node_name, state_diff) tuples and assert the hash is stable across replays. A drift means the graph is non-deterministic in a way it should not be (an unsignalled temperature > 0, a tool call that depends on wall-clock time, a parallel reduce that did not stabilise ordering).

The hash check costs nothing per replay and catches a class of bug that ages into a production flake. Without checkpoint replay you are guessing whether a bug reproduces. With it you are running an experiment.

Building a regression set from checkpoints

Synthetic test inputs reflect the test author’s assumptions. Sample real production checkpoints instead.

  1. Filter spans where an eval score dropped below threshold.
  2. Group by transition shape: cluster on the failing node’s gen_ai.agent.graph.node_id and the branch its routing span returned.
  3. Promote representative checkpoints from each cluster, paired with the expected downstream trajectory annotated by a reviewer.
  4. Tag with metadata: which release shipped the regression, which user segment, which rubric dropped.
  5. Re-run on every PR by replaying the saved checkpoints. CI gate fails the PR if any rubric falls below its baseline threshold.

Every incident becomes a replayable test case. For threshold-calibration cadence: LLM evaluation playbook for 2026.

Closing the loop with Error Feed and prompt optimization

Error Feed reads a sample of an Observe project’s traces, decides what went wrong in each, and groups traces that failed the same way into one issue. Each issue names a fix layer (Prompt, Tools, Orchestration, or Guardrails) and links the traces, spans, and sessions behind it. A root cause analysis returns a written root cause and a proposed fix.

Issue shapes to expect on a LangGraph project:

  • “Retry loop on tool X.” Ambiguous tool description or a node prompt that does not acknowledge tool failure.
  • “Wrong conditional edge on date-range queries.” A router prompt that does not handle a specific state shape.
  • “Reducer drops constraint after parallel fan-in.” A reducer function bug, not a prompt.
  • “Sub-graph silently returned partial state.” A sub-graph key the parent schema does not declare, which LangGraph drops.
  • “Checkpoint replay diverges from production trajectory.” Hidden non-determinism (wall-clock, unsignalled temperature, unstable parallel ordering).

Once an issue points at a node or router prompt, tune that prompt with the agent-learning-kit prompt optimizers, scored on your evals: RandomSearchOptimizer, BayesianSearchOptimizer, MetaPromptOptimizer, ProTeGi, GEPAOptimizer, and PromptWizardOptimizer. Give each node or router prompt its own run, and EarlyStoppingConfig stops a run that stalls or hits a score threshold. Background: automated optimization for agent systems.

How Future AGI ships this for LangGraph

  • agent-learning-kit (Apache-2.0). pip install agent-learning-kit; evaluate() runs local metrics such as task_completion and groundedness, cloud Turing templates such as evaluate_function_calling, or your own LLM-as-judge rubric, in CI.

  • traceAI (Apache-2.0). LangChainInstrumentor traces LangGraph nodes, routing functions, tools and LLM calls, with EvalTag server-side evals.

  • Future AGI Platform. Custom evals, continuous eval tasks on Observe spans, and Error Feed with five error groups, twenty categories and four fix layers.

  • Optimizers in agent-learning-kit. Six prompt optimizers scored on your evals.

  • Agent Command Center. A self-hostable, OpenAI-compatible gateway for the LLM calls under every node, with routing, failover, caching, and guardrails. Future AGI is SOC 2 Type II certified.

If your LangGraph is a five-node linear pipeline with no conditional edges, a single TaskCompletion rubric is enough. The four-layer stack earns its weight on graphs with real branching, retries, sub-graphs, or long-running checkpointed state. Framework-choice background: CrewAI vs LangGraph vs AutoGen and the best LangGraph alternatives.

What to do this week

One graph, end to end. Five steps.

  1. Install fi-instrumentation-otel and traceAI-langchain. Call register() and LangChainInstrumentor().instrument() in process bootstrap. Verify node spans carry thread_id.
  2. Turn on the checkpointer if you have not. Use durability="sync" or "async". The replay loop is only useful if the checkpoints exist.
  3. Add EvalTags for EVALUATE_FUNCTION_CALLING, TASK_COMPLETION, and the routing_accuracy and state_diff_correctness custom evals.
  4. Open Error Feed on the Observe project after a week of traffic, once its sampling rate is above 0. Promote the failing traces behind two issues into a regression set built from checkpoints, not synthetic inputs. Replay each checkpoint in CI on every PR.
  5. Run the agent-learning-kit optimizers on the regression set with BayesianSearchOptimizer and EarlyStoppingConfig. Tune the lowest-scoring router prompt as a single study target. Verify the winner does not regress any of the four layers.

Back to the agent from the opening: a 0.93 TaskCompletion score hid a router on the expensive branch, a retry loop, and a reducer that dropped a constraint. Per-node rubrics catch the reducer, the routing rubric catches the branch, and a replayed checkpoint shows the fix holds before the next deploy.

Trace your LangGraph agent and score it in Future AGI

Sources

Frequently Asked Questions

What is the LangGraph evaluation framework used for?

It is used to check that a LangGraph agent reaches the right answer through the right path, scoring final outputs, node state updates, routing decisions, and replayed checkpoints. That catches what a correct final answer hides: a router on the expensive branch, a retry loop, or a reducer that drops a constraint.

How do you evaluate a LangGraph agent?

Evaluate a LangGraph agent by tracing every node, scoring each node's input state and returned update, scoring every routing decision, and replaying saved checkpoints in CI. In Future AGI, traceAI records nodes and routing functions as spans, and custom LLM-as-judge evals score them in CI or on live traffic.

How do you test LangGraph agents?

Test LangGraph agents by saving the checkpoints of failing production runs and replaying them on every pull request, scoring the downstream path with your rubrics. Replay re-executes the nodes after the checkpoint, so pin temperature and stub external tools when a test needs the same path every time.

What is LangGraph trajectory evaluation?

It scores the path a LangGraph agent took, meaning the nodes it visited, the tools it called, and their order, against a reference path or an LLM judge. The agentevals package ships graph trajectory evaluators for strict matching and LLM-as-judge scoring, while transition rubrics show which node broke.

Which metrics should you track to evaluate a LangGraph agent?

Track routing accuracy per decision node, state-update correctness on high fan-out nodes, node input contracts, and checkpoint-replay trajectory matches next to task completion. Per-node scores point at the router prompt or reducer to fix, while one end-to-end score only says something broke somewhere in the graph.
Related Articles
View all