Articles

AI Agent Evaluation Metrics for Tool Calls, Plans and Outcomes

AI agent evaluation metrics for tool selection, arguments, step efficiency, plans, recovery and handoffs, with trace inputs, failure modes and a scored example.

· 18 min read
ai-agent-evaluation agent-evaluation-metrics llm-agents tool-calling multi-agent-systems agent-trajectory python
A transit map of three runs of one expense task between lookup_receipt and submit_expense, with one line detouring through flag_for_review and one passing a timed out call, plus each run's step efficiency score.
Table of Contents

An expense agent gets a plain request: file taxi receipt R-88 as a 42.50 travel expense. Its final answer reads “expense submitted”, and the output check gives it 1.0. Then you open the trace. Between the lookup and the submit sits a flag_for_review call, which put the receipt in a human reviewer’s queue for no reason.

AI agent evaluation metrics read the trajectory, the ordered list of tool calls behind that answer. A grader that sees only the final reply cannot see the path, so the wrong tool, a retry after a timeout and a wasted step all pass. The answer was right, and the run still cost a reviewer’s time.

This post gives one reference card per metric, a scored synthetic fixture built around receipt R-88, and a way to aggregate the scores and pick a small set for your agent. It is narrower than our guide to evaluating AI agents end to end.

TL;DR

  • AI agent evaluation metrics score the path an agent took through its tool calls, as well as the answer it returned.
  • Task completion, tool selection, argument correctness, step efficiency, plan adherence, recovery and handoff quality each read different fields from the trace.
  • Most of these run as plain code over the trace. An LLM judge covers open-ended results and plans written in prose.
  • On three synthetic runs of one expense task, task completion and tool selection both gave the wrong-tool run 1.0. An unexpected-tool count of 1 named its extra call.
  • Step efficiency in agent-learning-kit charges for the extra step, the repeated call and the timed-out call, so the recovered run scored 0.6667 and the wrong-tool run 0.8667. Read it next to recovery.
  • Report each metric per run, then per task set, and repeat each task to get pass^k, the chance that all k tries pass.

What AI agent evaluation metrics measure that LLM metrics miss

LLM agent evaluation metrics score the steps an agent takes as well as the text it returns. Generic LLM metrics read one prompt and one response. Agent evals read the trace: which tools were called, with which arguments, in what order, which calls failed, and whether the task finished.

Faithfulness, answer relevance and BLEU still apply to the text an agent writes, and our guide to LLM evaluation metrics covers them. An agent run is a loop of choices, though: pick a tool, fill its arguments, read the result, decide the next step.

The failures this post covers happen inside that loop, where a reply grader never looks. Seven metrics cover them: task completion, tool selection, argument correctness, step efficiency, plan adherence, recovery after a failed call and handoff quality. Each reads its own trace fields and fails in its own way, and the AI agent evaluation metrics card below lines them up.

MetricWhat it measuresTrace inputsCode or LLM judgeMain failure mode
Task completionDid the run reach the task’s goalFinal step flag, final and expected result, success criteriaCode heuristic; judge for open-ended resultsScores the end state, so a wrong path with a right result passes
Tool selectionNeeded tools called, valid names, calls succeededTool name and success per call; required and available toolsCodeAn extra valid tool costs nothing
Argument correctnessParameter names and values per callArguments per call; expected callCode; judge when the value depends on contextA number sent as a string does not match
Step efficiencySteps used against steps neededStep count, repeated calls, failed callsCodeCharges a correct retry as waste
Plan adherenceExecuted steps against the agent’s own planPlanning step output, executed sequenceCode for a tool list; judge for proseNo plan step, nothing to compare
Recovery after a failed callRun back on track after a failureError flags, later calls, final stepCodeCounts blind retries; misses unlisted fallbacks and escalations
Handoff qualityRouting, context passed, bounces, child resultAgent name, sender name, receiver outcomeCode; schema check or judge for contextBlame lands on the receiver when the sender dropped context

The trace fields every agent metric reads

Every metric on the card reads a few fields per step: the tool name, the arguments, the result, whether the call succeeded or raised an error, and whether the step was the final one. Multi-agent metrics add two per agent. They are the agent’s own name and the name of the agent that sent it the task.

traceAI, Future AGI’s tracing library for AI applications, is built on OpenTelemetry and records these as span attributes. Tool calls carry gen_ai.tool.name, gen_ai.tool.call.arguments, gen_ai.tool.call.result and error.type. Agents carry gen_ai.agent.name or graph.node.id, depending on the integration. In the OpenAI Agents SDK integration, graph.node.parent_id names the sender.

Outcome metrics versus trajectory metrics

Outcome metrics check the end state: was the expense filed with the right amount. Trajectory metrics check the route: which tools ran, in what order, and how many steps it took. An agent can pass the first and fail the second, so score both on every run.

The tau-bench paper is a clean reference for outcome grading. Its authors score a run by comparing the database state at the end of a conversation with the annotated goal state. A polite closing message cannot change a database row.

Trajectory metrics work two levels below the outcome: single tool calls, and the plan and trajectory they form. When the outcome check fails, they point at the step that broke. When it passes, they catch runs that took a costly detour, like the flag_for_review call on receipt R-88.

Three-level pyramid with tool call metrics at the base, plan and trajectory metrics in the middle and task completion at the top, each level linked to the trace inputs it reads

Task completion

Task completion asks whether the run reached the goal the task set. It reads the final step flag, the final result, the expected result and the success criteria. In agent-learning-kit, the local task_completion metric is a heuristic: 0.2 for a final step, 0.5 times the result match, and 0.3 times the share of criteria met.

For open-ended results, pass augment=True with a model, and the kit runs the heuristic first and then lets the model refine it. On our fixture, all three runs scored 1.0, the wrong-tool run included, because its end state was right. The metric scores the end state, so pair it with a trajectory metric.

When trajectory scoring misleads

Trajectory checks can mislead too. Our all-or-nothing ordered_sequence check passes only when the tool names equal lookup_receipt then submit_expense exactly, and it gave the recovered run 0.0. Retrying the timed-out lookup was the right move, yet the check scored it like the wrong tool.

Valid alternative paths hit the same wall. An agent that calls categorize_expense before submitting may be doing careful work, and a strict sequence match still fails it. Use unordered or subset matching where order does not matter, and keep strict order for steps that depend on each other.

A composite averages its components. The kit’s trajectory_score weighs task completion at 0.4, step efficiency at 0.3 and tool selection at 0.3, so the wrong-tool run scored 0.96. Read the component scores before the blend to see which step cost the points.

Tool selection and argument correctness metrics

Tool selection asks whether the agent called the tools the task needs and only those. Argument correctness asks whether each call carried the right parameter names and values. Both run as code against an expected call list, and they fail in different ways, so score them separately.

For more depth on scoring tool calls, see our post on evaluating tool-calling agents.

Tool selection accuracy

The agent-learning-kit tool_selection_accuracy metric combines required-tool coverage weighted 0.4, valid tool names weighted 0.3 and call success rate weighted 0.3. Coverage checks that each required tool appears. Validity checks each called name against the available tools, and success rate is the share of calls without an error.

On the fixture, the clean run scored 1.0 and the recovered run 0.9, because one of its three calls timed out. The wrong-tool run also scored 1.0. Its flag_for_review call went to an available tool, and both required tools were present.

The metric rewards coverage and validity, and an extra valid call touches neither. Pair it with an unexpected-tool count, the number of calls to tools outside the expected list. On the wrong-tool run that count was 1, and it points straight at flag_for_review.

Argument correctness

The kit’s function_call_accuracy splits a call’s score into the tool name at 0.4, parameter presence at 0.3 and value match at 0.3. We scored the submit_expense call on each run. All three got 1.0, since each carried id R-88, amount 42.5 and category travel.

Type drift is the failure. In the default mode an int and a float compare by value and strings compare case-sensitively, but a number sent as a string does not match a number. The same submit call with the amount sent as the string “42.50” against an expected 42.5 scored 0.9000 in our run.

Normalise types before scoring when your agent serialises arguments as strings, so “42.50” and 42.5 compare as the same amount. When the right value depends on context, such as a category inferred from receipt text, use an LLM judge that reads the call next to that context.

Step efficiency, plan adherence and recovery metrics

Step efficiency compares the steps an agent used with the steps the task needed. Plan adherence checks whether the agent did what its own plan said. Recovery checks what happened after a tool call failed. All three read the step list, and two of them run without any model.

Step efficiency

The plain version divides expected steps by actual steps and caps the result at 1.0. Both three-step runs scored 0.6667 on it. The kit’s step_efficiency blends 0.4 for the step ratio against the task’s max_steps, 0.3 for the nonredundancy score and 0.3 for the call-success score. Nonredundancy is 1 minus repeated calls divided by steps; call success is 1 minus failed calls divided by calls. A repeated call uses the same tool and arguments as an earlier call.

It measures the cost of each step by design, so the recovered run pays three times. The extra step lowers the ratio, the retry counts as redundant because it repeats the failed call’s tool and arguments, and the timed-out call counts as failed. The wrong-tool run pays on the ratio alone: 0.8667, against 0.6667.

The weighted score is:

step_efficiency = 0.4 × step ratio + 0.3 × nonredundancy + 0.3 × call success

RunStep ratioNonredundancyCall successStep efficiency
wrong_tool2/3110.8667
failed_then_recovered2/32/32/30.6667

The second failure mode sits in the expected count. A hard task with a short expected path punishes careful agents that check before they write. Set the expected step count per task, and read step efficiency next to recovery so a correct retry does not look like waste.

Plan adherence

Plan adherence needs an explicit plan in the trace, usually a planning step’s output. When the plan is a list of tool names, compare it with the executed sequence in code. When it is prose, an LLM judge reads it. Plan quality, whether the plan was any good, is a separate judge question.

Say the expense agent plans lookup_receipt then submit_expense, then calls flag_for_review between them. A list comparison marks one step off plan. Our fixture has no planning step, so we report no score. When an agent writes no plan, mark the metric not applicable and leave that run out of the mean.

Recovery after a failed tool call

Our recovery check passes when every failed call is followed by a successful call to the same tool and the run ends on a final step. The recovered run scored 1.0, and the two runs with no failure are not applicable. Blind retries also count under this rule, so cap the retries per tool.

A same-tool check also misses a fallback to a different tool or an explicit escalation, because it matches on the tool name. List each tool’s allowed fallbacks, and count a successful fallback or an escalation step as recovered. Step efficiency already deducts for the retry and the failed call, so do not subtract again.

Multi agent evaluation metrics for handoffs

A handoff passes a task from one agent to another, and the receiver can only work with what the sender passed. Four checks cover it: did the sender pick the right agent, did the task carry the context the receiver needed, did the task bounce back and forth, and did the child task succeed.

We call them routing accuracy, context carried, bounce count and child task success. Routing accuracy compares the chosen agent with a labelled expected agent, in code. Context carried is a schema check on required fields, such as a receipt id and an amount, or an LLM judge when the context is free text.

Bounce count is code over the chain of sender names. Child task success is an outcome metric on the receiver. When the sender drops context, the receiver’s failure looks like its own fault, so read the sender checks before you blame the receiver.

Suppose an intake agent hands receipt R-88 to a payments agent without the amount. Routing accuracy passes, because payments was the right agent, and context carried fails. The payments agent’s failed submit traces back to the sender. Our fixture has a single agent, so this example carries no score.

Reading handoffs from the trace

traceAI’s OpenAI Agents SDK instrumentation records each handoff as a tool-kind span named handoff to <agent>. On the receiving agent’s span it writes the agent’s name to graph.node.id and the sending agent’s name to graph.node.parent_id. The parent field holds a name, not a span id.

Several other integrations record gen_ai.agent.name instead. Bounce count reads the chain of sender names, so a task that moves intake, payments, intake, payments has bounced. Routing accuracy reads the first receiver against the label, and child task success reads the receiver’s final step.

Scoring three synthetic trajectories on one expense task

We wrote three synthetic trajectories for one expense task and scored them with agent-learning-kit metrics and four plain checks. Tool selection checks that the needed tools were called, so the wrong-tool run scored 1.0 and an unexpected-tool count of 1 named its extra call. Step efficiency charged the retry and timeout, so the recovered run scored lowest.

The script was tested end to end with Python 3.12.13 and agent-learning-kit 0.3.7 (latest on PyPI as of 2026-11-12), with no API key or network call. The synthetic fixture has one task, four available tools and three runs: clean, wrong tool, and a lookup that timed out, then succeeded on retry.

Five metrics come from the library: task_completion, step_efficiency, tool_selection_accuracy, trajectory_score and function_call_accuracy. Four are plain code in the script: ordered_sequence, unexpected_tools, own_step_efficiency and recovery. The table below copies the printed per-run scores, with n/a where a metric does not apply.

metriccleanwrong_toolfailed_then_recovered
task_completion1.00001.00001.0000
step_efficiency1.00000.86670.6667
tool_selection_accuracy1.00001.00000.9000
trajectory_score1.00000.96000.8700
function_call_accuracy1.00001.00001.0000
ordered_sequence1.00000.00000.0000
unexpected_tools010
own_step_efficiency1.00000.66670.6667
recoveryn/an/a1.0000

Read the rows by what each metric measures. Tool selection covers required tools, so the extra valid call kept full marks. The sequence check and both step efficiency scores marked it down, but only unexpected_tools named the call. Recovery gives the retry run 1.0 where step efficiency gives it 0.6667, below the 0.8667 of the wrong-tool run.

The kit’s trajectory_score carries step efficiency and tool selection at 0.3 each, and both charge the recovered run for its timed-out call, which gives 0.87 against 0.96 for the wrong-tool run. The script also prints a format-drift line: the submit call with the amount as the string “42.50” scores 0.9000 on function_call_accuracy.

The fixture and the checks

The task carries a description, required_tools (lookup_receipt and submit_expense), max_steps of 2 and one success criterion, “expense submitted”. Each step has a step_number, a tool_calls list whose entries hold name, arguments, success and error, plus is_final and observation. The kit’s metrics read these fields as they are.

The plain checks take a few lines each. The ordered_sequence check compares tool names with the expected list, and unexpected_tools counts calls outside it. own_step_efficiency divides expected steps by actual steps, and recovery looks for a later successful call to the same tool. Here is score_trajectories.py and its printed output.

"""Score three synthetic agent trajectories per run and per metric.

Metrics from agent-learning-kit (local, no API key, no network):
  tool_selection_accuracy, trajectory_score, task_completion, step_efficiency,
  function_call_accuracy.
Plain deterministic checks written here: ordered_sequence, unexpected_tools,
own_step_efficiency, recovery.

The trajectories are synthetic fixtures written for this script, not
production traffic.
"""
import json
from fi.evals import evaluate

TASK = {
    "description": "Submit the taxi receipt R-88 as a 42.50 travel expense",
    "required_tools": ["lookup_receipt", "submit_expense"],
    "max_steps": 2,
    "success_criteria": ["expense submitted"],
}
AVAILABLE = ["lookup_receipt", "categorize_expense", "submit_expense", "flag_for_review"]
EXPECTED_SEQUENCE = ["lookup_receipt", "submit_expense"]
EXPECTED_SUBMIT = {"name": "submit_expense",
                   "arguments": {"id": "R-88", "amount": 42.5, "category": "travel"}}
FINAL = "expense submitted"


def step(n, name, args, ok=True, final=False, obs=None):
    call = {"name": name, "arguments": args, "success": ok,
            "error": None if ok else "timeout"}
    return {"step_number": n, "tool_calls": [call], "is_final": final, "observation": obs}


SUBMIT = {"id": "R-88", "amount": 42.5, "category": "travel"}

# synthetic fixture: three hand-written trajectories
RUNS = {
    "clean": [
        step(1, "lookup_receipt", {"id": "R-88"}),
        step(2, "submit_expense", SUBMIT, final=True, obs=FINAL),
    ],
    "wrong_tool": [
        step(1, "lookup_receipt", {"id": "R-88"}),
        step(2, "flag_for_review", {"id": "R-88"}),
        step(3, "submit_expense", SUBMIT, final=True, obs=FINAL),
    ],
    "failed_then_recovered": [
        step(1, "lookup_receipt", {"id": "R-88"}, ok=False),
        step(2, "lookup_receipt", {"id": "R-88"}),
        step(3, "submit_expense", SUBMIT, final=True, obs=FINAL),
    ],
}


def calls(run):
    return [c for s in run for c in s["tool_calls"]]


def ordered_sequence(run):
    """1.0 when the tool names equal the expected sequence exactly."""
    return 1.0 if [c["name"] for c in calls(run)] == EXPECTED_SEQUENCE else 0.0


def unexpected_tools(run):
    """Count of calls to tools outside the expected sequence."""
    return sum(c["name"] not in EXPECTED_SEQUENCE for c in calls(run))


def own_step_efficiency(run):
    """expected steps divided by actual steps, capped at 1.0."""
    return min(1.0, len(EXPECTED_SEQUENCE) / len(run))


def recovery(run):
    """None when no call failed. Else 1.0 if every failed call was followed by a
    successful call to the same tool and the run ended with a final step."""
    cs = calls(run)
    failed = [i for i, c in enumerate(cs) if not c["success"]]
    if not failed:
        return None
    fixed = all(any(d["name"] == cs[i]["name"] and d["success"] for d in cs[i + 1:])
                for i in failed)
    return 1.0 if fixed and run[-1]["is_final"] else 0.0


def submit_args_score(run):
    """function_call_accuracy on the submit_expense call (name, parameters, values)."""
    sub = [c for c in calls(run) if c["name"] == "submit_expense"]
    return call_score(sub[0])


def call_score(call):
    got = {"name": call["name"], "arguments": call["arguments"]}
    r = evaluate("function_call_accuracy", response=json.dumps(got),
                 expected_response=json.dumps(EXPECTED_SUBMIT))
    return r.score


def kit_scores(run):
    out = {}
    for m in ["task_completion", "step_efficiency", "tool_selection_accuracy", "trajectory_score"]:
        r = evaluate(m, trajectory=run, task=TASK, available_tools=AVAILABLE,
                     final_result=FINAL, expected_result=FINAL)
        assert r.status == "completed", (m, r.status, r.error)
        out[m] = r.score
    return out


def fmt(x, nd=4):
    return "n/a" if x is None else f"{x:.{nd}f}"


def main():
    print("SYNTHETIC FIXTURE: three hand-written trajectories, one expense task")
    cols = ["task_completion", "step_efficiency", "tool_selection_accuracy", "trajectory_score",
            "function_call_accuracy", "ordered_sequence", "unexpected_tools",
            "own_step_efficiency", "recovery"]
    rows = {}
    for name, run in RUNS.items():
        k = kit_scores(run)
        rows[name] = {**k,
                      "function_call_accuracy": submit_args_score(run),
                      "ordered_sequence": ordered_sequence(run),
                      "unexpected_tools": unexpected_tools(run),
                      "own_step_efficiency": own_step_efficiency(run),
                      "recovery": recovery(run)}
    w = max(len(c) for c in cols)
    header = f"{'metric':<{w}}  " + "  ".join(f"{n:>21}" for n in RUNS)
    print(header)
    print("-" * len(header))
    for c in cols:
        cells = []
        for n in RUNS:
            v = rows[n][c]
            cells.append(f"{(str(v) if c == 'unexpected_tools' else fmt(v)):>21}")
        print(f"{c:<{w}}  " + "  ".join(cells))
    print()
    print("per task set mean over the 3 runs")
    for c in cols:
        vals = [rows[n][c] for n in RUNS if rows[n][c] is not None]
        print(f"{c:<{w}}  {sum(vals) / len(vals):.4f}  (n={len(vals)})")
    # the point of the fixture: the kit's tool_selection_accuracy does not see the
    # extra valid tool, the plain checks do
    assert rows["wrong_tool"]["tool_selection_accuracy"] == 1.0
    assert rows["wrong_tool"]["unexpected_tools"] == 1
    assert rows["wrong_tool"]["ordered_sequence"] == 0.0
    assert rows["failed_then_recovered"]["recovery"] == 1.0
    assert rows["clean"]["recovery"] is None
    # format drift: the amount sent as the string "42.50" instead of the number 42.5
    drift = call_score({"name": "submit_expense",
                        "arguments": {**SUBMIT, "amount": "42.50"}})
    print()
    print(f"function_call_accuracy with amount \"42.50\" (string) vs 42.5: {fmt(drift)}")
    assert drift < 1.0
    print()
    print("fixture expectations hold")


if __name__ == "__main__":
    main()
SYNTHETIC FIXTURE: three hand-written trajectories, one expense task
metric                                   clean             wrong_tool  failed_then_recovered
--------------------------------------------------------------------------------------------
task_completion                         1.0000                 1.0000                 1.0000
step_efficiency                         1.0000                 0.8667                 0.6667
tool_selection_accuracy                 1.0000                 1.0000                 0.9000
trajectory_score                        1.0000                 0.9600                 0.8700
function_call_accuracy                  1.0000                 1.0000                 1.0000
ordered_sequence                        1.0000                 0.0000                 0.0000
unexpected_tools                             0                      1                      0
own_step_efficiency                     1.0000                 0.6667                 0.6667
recovery                                   n/a                    n/a                 1.0000

per task set mean over the 3 runs
task_completion          1.0000  (n=3)
step_efficiency          0.8445  (n=3)
tool_selection_accuracy  0.9667  (n=3)
trajectory_score         0.9433  (n=3)
function_call_accuracy   1.0000  (n=3)
ordered_sequence         0.3333  (n=3)
unexpected_tools         0.3333  (n=3)
own_step_efficiency      0.7778  (n=3)
recovery                 1.0000  (n=1)

function_call_accuracy with amount "42.50" (string) vs 42.5: 0.9000

fixture expectations hold

Aggregating agent scores per run and per task set

Each metric stays its own number per run, then gets averaged per task set, leaving out runs where it does not apply. Across the three runs, recovery averages 1.0 over one run, not three. Repeat each task several times as well, because a single pass hides flaky agents.

Keep a vector per run, one number per metric, instead of one blended score. Our task-set means were 0.8445 for step_efficiency, 0.9667 for tool_selection_accuracy and 0.3333 for ordered_sequence. Recovery’s 1.0000 rests on n=1, since two runs had no failure, so print n beside each mean, as the script does.

The tau-bench authors define pass^k as the chance that all k independent trials of a task succeed, averaged across tasks. If a task runs n times and c runs succeed, the estimate is C(c,k) divided by C(n,k). For any task with a failed trial, that estimate falls toward zero as k grows.

The original repo now points to tau2-bench for the latest version.

Latency targets and the SLOs built on these numbers are a separate job, covered in our post on AI agent reliability metrics.

A minimal metric set by agent type

Build the set from pairs. Each metric sits next to the check that covers its blind spot. Our run showed three: tool selection with the unexpected-tool count (1.0 and 1 on the wrong-tool run), step efficiency with recovery (0.6667 and 1.0 on the retry run), and task completion with one trajectory metric.

Then pick pairs by the failure your agent faces. One lookup and one write risk wrong argument types, so pair argument correctness with a type check on values. Workflows over many tools add or drop calls, so use tool selection plus the unexpected-tool count. If the agent writes a plan, add plan adherence, excluded when no plan step exists.

Timeouts call for step efficiency plus recovery, with fallbacks listed per tool. With several agents, score routing accuracy plus context carried, then child task success, because a dropped handoff starts with the sender. Add a metric only when a run fails in a way the current set cannot name.

Running AI agent evaluation metrics on traced spans in Future AGI

Future AGI runs evals on the spans traceAI records. In Observe you attach preset or custom evals to a project, pick spans, traces or sessions, and run them on historic traffic or continuously on new spans. Scores land as a new column in the trace table, and each span shows its own score.

The docs on evals in Observe describe two run types. The Historical type runs over existing spans that match your filters, then completes. The Continuous type scores new spans as they arrive and stays active, so the checks from a fixture keep scoring production traffic.

When you set up an eval task, you pick Spans, Traces or Sessions as the level it scores. Variable mapping ties each eval input to a span attribute path. Paths use dot notation, with numeric positions such as .0 for array items.

The eval types page covers Code Eval, deterministic Python or JavaScript that calls no model. It also covers LLM-as-Judge, which applies a template’s criteria in one pass. Use Code Eval for unexpected tools and recovery, and LLM-as-Judge for plan quality and context carried.

The built-in trajectory match eval is a code-based check that compares an actual and an expected action list, each a JSON array of names. It runs in strict, unordered, subset or superset mode. Unordered mode treats both lists as sets and scores their Jaccard overlap.

For local runs, agent-learning-kit on GitHub ships tool_selection_accuracy, trajectory_score and function_call_accuracy among its 20+ local deterministic metrics. You call them through from fi.evals import evaluate with no API key. That is how the fixture here was scored.

From the fixture to production spans

The plain checks the script ran, unexpected tools, recovery and our own step efficiency, become Code Eval custom evals. For order, trajectory_match in strict mode scores the length of the matching prefix against the longer list. That gives partial credit where our all-or-nothing check printed 0.0.

Superset mode lowers the score when a tool outside the expected list appears. A retry of the same tool leaves it unchanged. That makes it the built-in counterpart of the unexpected-tool count. In superset mode, flag_for_review would cost the wrong-tool run, and the second lookup would not.

Catching the wrong tool before a receipt reaches review

One extra tool call put the receipt in a reviewer’s queue, and the run still scored 1.0 on its answer. An unexpected-tool count of 1 names that call; an all-or-nothing sequence check flags the run but also fails a correct retry. Pick the metrics your agent type needs and score every run.

On receipt R-88, task completion was right about the end state, and the unexpected-tool count put flag_for_review on the record. On the other run, recovery and step efficiency showed what the retry cost. Each number pointed at a different step, so keep them separate.

Start with the pairs for your agent type. Record a handful of runs per task, and score each run with the same checks. Once the numbers make sense on recorded runs, attach the checks to traced spans, and the next stray review call shows up the day it happens.

Frequently Asked Questions

What are common AI agent evaluation metrics?

Common AI agent evaluation metrics are task completion, tool selection, argument correctness, step efficiency, plan adherence and recovery after a failed call. Multi-agent systems add handoff checks. Each metric reads different fields from the trace, so report them as separate numbers rather than one blended score.

What are evals in agentic AI?

Evals in agentic AI are automated checks that score an agent's run, covering the final result and the tool calls, arguments and steps that produced it. Step efficiency reads the trace as well as the reply, so a wasted tool call costs points even when the answer is correct. Each eval returns a score you can track per run.

What are the types of evals?

For agents, the three types of evals are code-based checks, LLM-as-a-judge scoring and human review of a small labelled sample of runs. The human slice calibrates the judge: label each sampled run pass or fail with a short reason, run the judge on the same runs, and rewrite the judge prompt wherever its verdict disagrees with the label.

What are multi agent evaluation metrics?

Multi agent evaluation metrics score the handoffs between agents as well as each agent's own steps and the final result. Check that the sender picked the right agent, that the task carried the needed context, that it did not bounce back and forth, and that the child task succeeded.

What is an agent evaluation trajectory?

An agent evaluation trajectory is the ordered record of the steps, tool calls, arguments and results an agent produced on one run. Metrics score it by comparing calls with an expected list, counting wasted or failed steps, and checking order. Accept valid alternative paths, or a correct retry looks like a failure.
Related Articles
View all