Agent Trace Evaluation With Code-Based Checks for Tool Calls

Your nightly eval job sends every agent trace to an LLM judge. Many of the judge’s questions are mechanical, like whether the agent called a tool that exists. The bill grows with traffic, and the same trace scores differently when you rerun it.

A schema check or an exact match settles those questions for almost nothing, and it gives the same answer every time. Agent trace evaluation works better with code-based checks in front of the judge, so the judge only sees the questions that need judgment.

TL;DR

• Score every agent trace with the cheapest check that can settle the question, and escalate only what is left.
• Rungs one to four are code: schema checks, function-call checks, trajectory matching, and partial-credit tool-call accuracy.
• Pick the trajectory mode on purpose, and read each library’s definition of subset and superset.
• Save the LLM judge for questions about meaning, such as groundedness.
• Run the same checks on live traces and as a CI gate.

Agent trace evaluation reads spans before it calls a judge

Every trace starts on rung one of a cost ladder and moves up only when a lower rung cannot answer the question. For mechanical questions, code-based checks replace the judge. Choosing between several valid tools still needs one.

Deterministic evals cost close to nothing and never drift between runs. A judge costs money on every call and can score the same trace differently on a rerun.

Our post on deterministic vs LLM-judge evals explains the trade-off, and the docs describe code evals, LLM judges, and agent evaluators the same way.

Rung one: schema checks on every tool span in the trace

A tool span records the tool name and the arguments the agent sent, which makes it the first place to check. If you need a refresher on what each record holds, see our explainer on spans vs traces.

Before any schema check, make sure the span holds the arguments. In the OpenTelemetry GenAI span conventions, still in development status, tool-call arguments and results are opt-in attributes, so turn capture on or rungs one, two, and four have nothing to read.

Then validate the arguments against the tool’s JSON schema, use regex on formatted fields like airport codes or IDs, and check equality on fixed values. Confirm that required fields are present and reject calls to tools that do not exist. These checks run in plain Python with no model call.

Turn on format checking, because JSON Schema treats "format": "date" as a note unless the validator enforces it. The worked example below shows the one-line switch.

Rung two: right function, valid parameters, exact or partial match

Tool call evaluation at this rung looks at one call at a time. First check that the agent called the right function, then that its arguments match the expected call. Report missing and extra calls as their own rows, so a skipped booking cannot hide behind a list of matches.

Then separate an exact match from a partial one. An exact match means the name and every argument equal the expected call, and a partial match means the name is right but an argument differs, such as the wrong date on a flight search. That distinction feeds the partial credit on rung four.

Normalize arguments before you compare them. Plain equality fails JFK against jfk, or a date against the same date with a time attached, so canonicalize case, dates, and units first.

Rung three: agent trajectory evaluation against an expected path

Agent trajectory evaluation compares the sequence of tool calls with an expected path, and the mode decides how much freedom the agent gets. Strict allows none, unordered ignores order, and subset and superset allow extra steps or skipped steps. The trajectory match modes define each one precisely.

The three looser modes compare sets, so duplicates and order are invisible to them, and only strict mode notices a repeated or reordered call.

Watch the mode names. Libraries disagree on which direction subset and superset mean, so check each library’s definition before trusting the label. The table below uses Future AGI’s definitions. Our guide to the four-layer eval stack for tool-calling agents puts these checks in a wider stack.

Rung four: tool-call accuracy with partial credit

Pass or fail hides near misses, so rung four gives partial credit. A call earns full credit when the name and arguments both match, and half credit when only the name matches. The total is divided by the longer of the two lists, so extra or missing calls cost points.

That last rule matters for agents that repeat themselves. A duplicate search makes the actual list longer, which lowers the score even when every expected call eventually happened. The docs describe tool call accuracy with partial credit with the same 1.0, 0.5, and longer-list scoring.

Rung five: an LLM judge only for questions about meaning

Code cannot settle questions about meaning. Is the final answer grounded in the tool results? Was it helpful, and was the tone right for your policy? Did the agent pick a sensible tool when several would have worked? A schema or an exact match has nothing to compare against there, so an LLM judge earns its cost.

Send the judge only the traces that cleared rungs one to four, and give it the whole trajectory, including tool results, along with the final answer. Traces that fail a lower rung go to triage instead. The judge bill then scales with the open questions alone.

Rungs two to four need a reference path, and open-ended agents often have several valid ones. When the path varies, keep rung one plus invariant checks, such as no forbidden tool and a step budget, and let the judge score the route. Refresh expected paths whenever a tool changes.

For tools with side effects, add a state check as well. After the run, query the booking table and confirm one booking exists with the right flight and date. That check is deterministic, and it passes a valid alternative path that trajectory matching would fail.

Worked example: one flight-booking trace through every rung

The expected path is simple: search flights, then book one. The actual trace searched twice with the wrong date, then booked with an extra seat field the tool does not accept.

The code below runs rungs one to four on that trace, using jsonschema (tested on 4.26.0, installed with pip install jsonschema) and the standard library.

from itertools import zip_longest
from jsonschema import Draft202012Validator

TOOL_SCHEMAS = {
    "search_flights": {
        "type": "object",
        "properties": {"origin": {"type": "string", "pattern": "^[A-Z]{3}$"},
                       "destination": {"type": "string", "pattern": "^[A-Z]{3}$"},
                       "date": {"type": "string", "format": "date"}},
        "required": ["origin", "destination", "date"],
        "additionalProperties": False,
    },
    "book_flight": {
        "type": "object",
        "properties": {"flight_id": {"type": "string"}},
        "required": ["flight_id"],
        "additionalProperties": False,
    },
}

def normalize(args):
    """Canonicalize before comparing. Uppercasing suits airport codes; add your own rules for dates, units, and case-sensitive IDs."""
    return {k: v.strip().upper() if isinstance(v, str) else v for k, v in (args or {}).items()}

def rung1_structure(calls):
    """Unknown tool or schema-invalid arguments. format_checker makes "format": "date" enforce real dates."""
    errors = []
    for c in calls:
        schema = TOOL_SCHEMAS.get(c["name"])
        if schema is None:
            errors.append(f"{c['name']}: tool does not exist")
            continue
        validator = Draft202012Validator(schema, format_checker=Draft202012Validator.FORMAT_CHECKER)
        errors += [f"{c['name']}: {e.message}" for e in validator.iter_errors(c["arguments"])]
    return errors

def rung2_function_calls(actual, expected):
    """Per position: right function name, and an exact match on name plus normalized arguments."""
    rows = []
    for got, want in zip_longest(actual, expected):
        if got is None or want is None:
            rows.append({"expected": want and want["name"], "actual": got and got["name"], "status": "missing" if got is None else "extra"})
            continue
        rows.append({"expected": want["name"], "name_match": got["name"] == want["name"],
                     "exact_match": got["name"] == want["name"] and normalize(got["arguments"]) == normalize(want["arguments"])})
    return rows

def rung3_trajectory(actual, expected, mode="strict"):
    """Set-based modes ignore duplicates and order; only strict sees them."""
    a, e = [c["name"] for c in actual], [c["name"] for c in expected]
    if mode == "strict":        # in-order prefix match, normalized by the longer list
        prefix = 0
        for x, y in zip(a, e):
            if x != y:
                break
            prefix += 1
        return prefix / max(len(a), len(e), 1)
    if mode == "unordered":     # Jaccard overlap of the two sets
        return len(set(a) & set(e)) / max(len(set(a) | set(e)), 1)
    if mode == "subset":        # every expected step happened
        return float(set(e) <= set(a))
    if mode == "superset":      # every step taken was expected
        return float(set(a) <= set(e))
    raise ValueError(mode)

def rung4_tool_call_accuracy(actual, expected):
    """1.0 for name and arguments, 0.5 for name only, paired by position, averaged over the longer list."""
    score = 0.0
    for got, want in zip(actual, expected):
        if got["name"] == want["name"]:
            score += 1.0 if normalize(got["arguments"]) == normalize(want["arguments"]) else 0.5
    return score / max(len(actual), len(expected), 1)

expected = [{"name": "search_flights", "arguments": {"origin": "SFO", "destination": "JFK", "date": "2026-11-02"}},
            {"name": "book_flight", "arguments": {"flight_id": "UA-512"}}]
actual = [{"name": "search_flights", "arguments": {"origin": "SFO", "destination": "JFK", "date": "2026-11-03"}},
          {"name": "search_flights", "arguments": {"origin": "SFO", "destination": "JFK", "date": "2026-11-03"}},
          {"name": "book_flight", "arguments": {"flight_id": "UA-512", "seat": "12A"}}]

print("rung 1:", rung1_structure(actual))     # extra 'seat' field on book_flight
print("rung 2:", rung2_function_calls(actual, expected))   # name only, then wrong name, then one extra call
for m in ("strict", "unordered", "subset", "superset"):
    print("rung 3", m, round(rung3_trajectory(actual, expected, m), 2))   # 0.33, 1.0, 1.0, 1.0
print("rung 4:", round(rung4_tool_call_accuracy(actual, expected), 2))   # 0.17

Rung one flags the extra seat field on book_flight. The wrong date is still a valid date, so the schema passes it.

Rung two catches the date instead: the first call has the right name but no exact match. The duplicate search pushes the booking out of position, so the second pair fails on name and the booking shows up as an extra call.

On rung three, strict scores 0.33, while unordered, subset, and superset all pass, which makes the mode choice decisive. Rung four scores 0.17, since only the first call earns half credit across three calls.

This code pairs calls by position for readability. Future AGI’s Tool Call Accuracy matches each call to its best unused expected call and scores this trace 0.33. Its Trajectory Match also scores strict 0.33 here, and its subset and superset modes return partial scores when steps are missing or unexpected.

Code surfaced three defects here without a judge call: the extra seat field, the wrong date, and the duplicate search. The trace failed, so it goes to triage, and it never reaches rung five.

Running agent trace evaluation on live traces and on every pull request

Run the same checks in two places. On live traffic, sample traces and run evals on spans, traces, or sessions. A broken tool schema can then surface as a falling score soon after the deploy, before users report it.

In CI, run the same checks on a fixed set of traces and call the SDK from your pipeline. Fail the pull request when the pass rate drops. Our posts on trace-native evaluation and how to instrument your agent with traceAI cover the tracing setup this depends on.

Code-based tool-call checks with Future AGI

Code Eval runs your own Python or JavaScript in a sandbox and calls no model. Local metrics such as json_schema, regex, contains, and equals, plus the function-calling metrics, run from the open-source ai-evaluation SDK, now developed in the Agent Learning Kit repo.

Install it with pip install ai-evaluation. The snippet below was tested on version 1.1.0 and runs locally with no API key.

from fi.evals import evaluate

actual = '{"name": "search_flights", "arguments": {"origin": "SFO", "destination": "JFK", "date": "2026-11-03"}}'
expected = '{"name": "search_flights", "arguments": {"origin": "SFO", "destination": "JFK", "date": "2026-11-02"}}'

print(evaluate("function_name_match", output=actual, expected_output=expected).score)        # 1.0
print(evaluate("function_call_exact_match", output=actual, expected_output=expected).score)  # 0.0

Trajectory Match and Tool Call Accuracy are built-in platform evals that use the modes and partial-credit scoring described above. Both run on live traces in Observe, the same way as the checks above. You can browse the evals and tracing code in the Future AGI repo.

Cutting the judge bill with code-based tool-call checks

Back to the nightly job. If it runs rungs one to four on every trace first, the judge only sees traces that passed them. The flight-booking trace never reaches the judge, because code surfaced all three defects first, and its structural scores come out the same on every rerun.

Start with one rung-one check on your most-used tool, add the other rungs over the next few releases, and your judge spend will track the open questions that remain.

Run your first code eval with the open-source SDK on today’s traces.