AI Agent Evaluation Metrics for Tool Calls, Plans and Outcomes
AI agent evaluation metrics for tool selection, arguments, step efficiency, plans, recovery and handoffs, with trace inputs, failure modes and a scored example.
Table of Contents
An expense agent gets a plain request: file taxi receipt R-88 as a 42.50 travel expense. Its final answer reads “expense submitted”, and the output check gives it 1.0. Then you open the trace. Between the lookup and the submit sits a flag_for_review call, which put the receipt in a human reviewer’s queue for no reason.
AI agent evaluation metrics read the trajectory, the ordered list of tool calls behind that answer. A grader that sees only the final reply cannot see the path, so the wrong tool, a retry after a timeout and a wasted step all pass. The answer was right, and the run still cost a reviewer’s time.
This post gives one reference card per metric, a scored synthetic fixture built around receipt R-88, and a way to aggregate the scores and pick a small set for your agent. It is narrower than our guide to evaluating AI agents end to end.
TL;DR
- AI agent evaluation metrics score the path an agent took through its tool calls, as well as the answer it returned.
- Task completion, tool selection, argument correctness, step efficiency, plan adherence, recovery and handoff quality each read different fields from the trace.
- Most of these run as plain code over the trace. An LLM judge covers open-ended results and plans written in prose.
- On three synthetic runs of one expense task, task completion and tool selection both gave the wrong-tool run 1.0. An unexpected-tool count of 1 named its extra call.
- Step efficiency in agent-learning-kit charges for the extra step, the repeated call and the timed-out call, so the recovered run scored 0.6667 and the wrong-tool run 0.8667. Read it next to recovery.
- Report each metric per run, then per task set, and repeat each task to get pass^k, the chance that all k tries pass.
What AI agent evaluation metrics measure that LLM metrics miss
LLM agent evaluation metrics score the steps an agent takes as well as the text it returns. Generic LLM metrics read one prompt and one response. Agent evals read the trace: which tools were called, with which arguments, in what order, which calls failed, and whether the task finished.
Faithfulness, answer relevance and BLEU still apply to the text an agent writes, and our guide to LLM evaluation metrics covers them. An agent run is a loop of choices, though: pick a tool, fill its arguments, read the result, decide the next step.
The failures this post covers happen inside that loop, where a reply grader never looks. Seven metrics cover them: task completion, tool selection, argument correctness, step efficiency, plan adherence, recovery after a failed call and handoff quality. Each reads its own trace fields and fails in its own way, and the AI agent evaluation metrics card below lines them up.
| Metric | What it measures | Trace inputs | Code or LLM judge | Main failure mode |
|---|---|---|---|---|
| Task completion | Did the run reach the task’s goal | Final step flag, final and expected result, success criteria | Code heuristic; judge for open-ended results | Scores the end state, so a wrong path with a right result passes |
| Tool selection | Needed tools called, valid names, calls succeeded | Tool name and success per call; required and available tools | Code | An extra valid tool costs nothing |
| Argument correctness | Parameter names and values per call | Arguments per call; expected call | Code; judge when the value depends on context | A number sent as a string does not match |
| Step efficiency | Steps used against steps needed | Step count, repeated calls, failed calls | Code | Charges a correct retry as waste |
| Plan adherence | Executed steps against the agent’s own plan | Planning step output, executed sequence | Code for a tool list; judge for prose | No plan step, nothing to compare |
| Recovery after a failed call | Run back on track after a failure | Error flags, later calls, final step | Code | Counts blind retries; misses unlisted fallbacks and escalations |
| Handoff quality | Routing, context passed, bounces, child result | Agent name, sender name, receiver outcome | Code; schema check or judge for context | Blame lands on the receiver when the sender dropped context |
The trace fields every agent metric reads
Every metric on the card reads a few fields per step: the tool name, the arguments, the result, whether the call succeeded or raised an error, and whether the step was the final one. Multi-agent metrics add two per agent. They are the agent’s own name and the name of the agent that sent it the task.
traceAI, Future AGI’s tracing library for AI applications, is built on OpenTelemetry and records these as span attributes. Tool calls carry gen_ai.tool.name, gen_ai.tool.call.arguments, gen_ai.tool.call.result and error.type. Agents carry gen_ai.agent.name or graph.node.id, depending on the integration. In the OpenAI Agents SDK integration, graph.node.parent_id names the sender.
Outcome metrics versus trajectory metrics
Outcome metrics check the end state: was the expense filed with the right amount. Trajectory metrics check the route: which tools ran, in what order, and how many steps it took. An agent can pass the first and fail the second, so score both on every run.
The tau-bench paper is a clean reference for outcome grading. Its authors score a run by comparing the database state at the end of a conversation with the annotated goal state. A polite closing message cannot change a database row.
Trajectory metrics work two levels below the outcome: single tool calls, and the plan and trajectory they form. When the outcome check fails, they point at the step that broke. When it passes, they catch runs that took a costly detour, like the flag_for_review call on receipt R-88.

Task completion
Task completion asks whether the run reached the goal the task set. It reads the final step flag, the final result, the expected result and the success criteria. In agent-learning-kit, the local task_completion metric is a heuristic: 0.2 for a final step, 0.5 times the result match, and 0.3 times the share of criteria met.
For open-ended results, pass augment=True with a model, and the kit runs the heuristic first and then lets the model refine it. On our fixture, all three runs scored 1.0, the wrong-tool run included, because its end state was right. The metric scores the end state, so pair it with a trajectory metric.
When trajectory scoring misleads
Trajectory checks can mislead too. Our all-or-nothing ordered_sequence check passes only when the tool names equal lookup_receipt then submit_expense exactly, and it gave the recovered run 0.0. Retrying the timed-out lookup was the right move, yet the check scored it like the wrong tool.
Valid alternative paths hit the same wall. An agent that calls categorize_expense before submitting may be doing careful work, and a strict sequence match still fails it. Use unordered or subset matching where order does not matter, and keep strict order for steps that depend on each other.
A composite averages its components. The kit’s trajectory_score weighs task completion at 0.4, step efficiency at 0.3 and tool selection at 0.3, so the wrong-tool run scored 0.96. Read the component scores before the blend to see which step cost the points.
Tool selection and argument correctness metrics
Tool selection asks whether the agent called the tools the task needs and only those. Argument correctness asks whether each call carried the right parameter names and values. Both run as code against an expected call list, and they fail in different ways, so score them separately.
For more depth on scoring tool calls, see our post on evaluating tool-calling agents.
Tool selection accuracy
The agent-learning-kit tool_selection_accuracy metric combines required-tool coverage weighted 0.4, valid tool names weighted 0.3 and call success rate weighted 0.3. Coverage checks that each required tool appears. Validity checks each called name against the available tools, and success rate is the share of calls without an error.
On the fixture, the clean run scored 1.0 and the recovered run 0.9, because one of its three calls timed out. The wrong-tool run also scored 1.0. Its flag_for_review call went to an available tool, and both required tools were present.
The metric rewards coverage and validity, and an extra valid call touches neither. Pair it with an unexpected-tool count, the number of calls to tools outside the expected list. On the wrong-tool run that count was 1, and it points straight at flag_for_review.
Argument correctness
The kit’s function_call_accuracy splits a call’s score into the tool name at 0.4, parameter presence at 0.3 and value match at 0.3. We scored the submit_expense call on each run. All three got 1.0, since each carried id R-88, amount 42.5 and category travel.
Type drift is the failure. In the default mode an int and a float compare by value and strings compare case-sensitively, but a number sent as a string does not match a number. The same submit call with the amount sent as the string “42.50” against an expected 42.5 scored 0.9000 in our run.
Normalise types before scoring when your agent serialises arguments as strings, so “42.50” and 42.5 compare as the same amount. When the right value depends on context, such as a category inferred from receipt text, use an LLM judge that reads the call next to that context.
Step efficiency, plan adherence and recovery metrics
Step efficiency compares the steps an agent used with the steps the task needed. Plan adherence checks whether the agent did what its own plan said. Recovery checks what happened after a tool call failed. All three read the step list, and two of them run without any model.
Step efficiency
The plain version divides expected steps by actual steps and caps the result at 1.0. Both three-step runs scored 0.6667 on it. The kit’s step_efficiency blends 0.4 for the step ratio against the task’s max_steps, 0.3 for the nonredundancy score and 0.3 for the call-success score. Nonredundancy is 1 minus repeated calls divided by steps; call success is 1 minus failed calls divided by calls. A repeated call uses the same tool and arguments as an earlier call.
It measures the cost of each step by design, so the recovered run pays three times. The extra step lowers the ratio, the retry counts as redundant because it repeats the failed call’s tool and arguments, and the timed-out call counts as failed. The wrong-tool run pays on the ratio alone: 0.8667, against 0.6667.
The weighted score is:
step_efficiency = 0.4 × step ratio + 0.3 × nonredundancy + 0.3 × call success
| Run | Step ratio | Nonredundancy | Call success | Step efficiency |
|---|---|---|---|---|
wrong_tool | 2/3 | 1 | 1 | 0.8667 |
failed_then_recovered | 2/3 | 2/3 | 2/3 | 0.6667 |
The second failure mode sits in the expected count. A hard task with a short expected path punishes careful agents that check before they write. Set the expected step count per task, and read step efficiency next to recovery so a correct retry does not look like waste.
Plan adherence
Plan adherence needs an explicit plan in the trace, usually a planning step’s output. When the plan is a list of tool names, compare it with the executed sequence in code. When it is prose, an LLM judge reads it. Plan quality, whether the plan was any good, is a separate judge question.
Say the expense agent plans lookup_receipt then submit_expense, then calls flag_for_review between them. A list comparison marks one step off plan. Our fixture has no planning step, so we report no score. When an agent writes no plan, mark the metric not applicable and leave that run out of the mean.
Recovery after a failed tool call
Our recovery check passes when every failed call is followed by a successful call to the same tool and the run ends on a final step. The recovered run scored 1.0, and the two runs with no failure are not applicable. Blind retries also count under this rule, so cap the retries per tool.
A same-tool check also misses a fallback to a different tool or an explicit escalation, because it matches on the tool name. List each tool’s allowed fallbacks, and count a successful fallback or an escalation step as recovered. Step efficiency already deducts for the retry and the failed call, so do not subtract again.
Multi agent evaluation metrics for handoffs
A handoff passes a task from one agent to another, and the receiver can only work with what the sender passed. Four checks cover it: did the sender pick the right agent, did the task carry the context the receiver needed, did the task bounce back and forth, and did the child task succeed.
We call them routing accuracy, context carried, bounce count and child task success. Routing accuracy compares the chosen agent with a labelled expected agent, in code. Context carried is a schema check on required fields, such as a receipt id and an amount, or an LLM judge when the context is free text.
Bounce count is code over the chain of sender names. Child task success is an outcome metric on the receiver. When the sender drops context, the receiver’s failure looks like its own fault, so read the sender checks before you blame the receiver.
Suppose an intake agent hands receipt R-88 to a payments agent without the amount. Routing accuracy passes, because payments was the right agent, and context carried fails. The payments agent’s failed submit traces back to the sender. Our fixture has a single agent, so this example carries no score.
Reading handoffs from the trace
traceAI’s OpenAI Agents SDK instrumentation records each handoff as a tool-kind span named handoff to <agent>. On the receiving agent’s span it writes the agent’s name to graph.node.id and the sending agent’s name to graph.node.parent_id. The parent field holds a name, not a span id.
Several other integrations record gen_ai.agent.name instead. Bounce count reads the chain of sender names, so a task that moves intake, payments, intake, payments has bounced. Routing accuracy reads the first receiver against the label, and child task success reads the receiver’s final step.
Scoring three synthetic trajectories on one expense task
We wrote three synthetic trajectories for one expense task and scored them with agent-learning-kit metrics and four plain checks. Tool selection checks that the needed tools were called, so the wrong-tool run scored 1.0 and an unexpected-tool count of 1 named its extra call. Step efficiency charged the retry and timeout, so the recovered run scored lowest.
The script was tested end to end with Python 3.12.13 and agent-learning-kit 0.3.7 (latest on PyPI as of 2026-11-12), with no API key or network call. The synthetic fixture has one task, four available tools and three runs: clean, wrong tool, and a lookup that timed out, then succeeded on retry.
Five metrics come from the library: task_completion, step_efficiency, tool_selection_accuracy, trajectory_score and function_call_accuracy. Four are plain code in the script: ordered_sequence, unexpected_tools, own_step_efficiency and recovery. The table below copies the printed per-run scores, with n/a where a metric does not apply.
| metric | clean | wrong_tool | failed_then_recovered |
|---|---|---|---|
| task_completion | 1.0000 | 1.0000 | 1.0000 |
| step_efficiency | 1.0000 | 0.8667 | 0.6667 |
| tool_selection_accuracy | 1.0000 | 1.0000 | 0.9000 |
| trajectory_score | 1.0000 | 0.9600 | 0.8700 |
| function_call_accuracy | 1.0000 | 1.0000 | 1.0000 |
| ordered_sequence | 1.0000 | 0.0000 | 0.0000 |
| unexpected_tools | 0 | 1 | 0 |
| own_step_efficiency | 1.0000 | 0.6667 | 0.6667 |
| recovery | n/a | n/a | 1.0000 |
Read the rows by what each metric measures. Tool selection covers required tools, so the extra valid call kept full marks. The sequence check and both step efficiency scores marked it down, but only unexpected_tools named the call. Recovery gives the retry run 1.0 where step efficiency gives it 0.6667, below the 0.8667 of the wrong-tool run.
The kit’s trajectory_score carries step efficiency and tool selection at 0.3 each, and both charge the recovered run for its timed-out call, which gives 0.87 against 0.96 for the wrong-tool run. The script also prints a format-drift line: the submit call with the amount as the string “42.50” scores 0.9000 on function_call_accuracy.
The fixture and the checks
The task carries a description, required_tools (lookup_receipt and submit_expense), max_steps of 2 and one success criterion, “expense submitted”. Each step has a step_number, a tool_calls list whose entries hold name, arguments, success and error, plus is_final and observation. The kit’s metrics read these fields as they are.
The plain checks take a few lines each. The ordered_sequence check compares tool names with the expected list, and unexpected_tools counts calls outside it. own_step_efficiency divides expected steps by actual steps, and recovery looks for a later successful call to the same tool. Here is score_trajectories.py and its printed output.
"""Score three synthetic agent trajectories per run and per metric.
Metrics from agent-learning-kit (local, no API key, no network):
tool_selection_accuracy, trajectory_score, task_completion, step_efficiency,
function_call_accuracy.
Plain deterministic checks written here: ordered_sequence, unexpected_tools,
own_step_efficiency, recovery.
The trajectories are synthetic fixtures written for this script, not
production traffic.
"""
import json
from fi.evals import evaluate
TASK = {
"description": "Submit the taxi receipt R-88 as a 42.50 travel expense",
"required_tools": ["lookup_receipt", "submit_expense"],
"max_steps": 2,
"success_criteria": ["expense submitted"],
}
AVAILABLE = ["lookup_receipt", "categorize_expense", "submit_expense", "flag_for_review"]
EXPECTED_SEQUENCE = ["lookup_receipt", "submit_expense"]
EXPECTED_SUBMIT = {"name": "submit_expense",
"arguments": {"id": "R-88", "amount": 42.5, "category": "travel"}}
FINAL = "expense submitted"
def step(n, name, args, ok=True, final=False, obs=None):
call = {"name": name, "arguments": args, "success": ok,
"error": None if ok else "timeout"}
return {"step_number": n, "tool_calls": [call], "is_final": final, "observation": obs}
SUBMIT = {"id": "R-88", "amount": 42.5, "category": "travel"}
# synthetic fixture: three hand-written trajectories
RUNS = {
"clean": [
step(1, "lookup_receipt", {"id": "R-88"}),
step(2, "submit_expense", SUBMIT, final=True, obs=FINAL),
],
"wrong_tool": [
step(1, "lookup_receipt", {"id": "R-88"}),
step(2, "flag_for_review", {"id": "R-88"}),
step(3, "submit_expense", SUBMIT, final=True, obs=FINAL),
],
"failed_then_recovered": [
step(1, "lookup_receipt", {"id": "R-88"}, ok=False),
step(2, "lookup_receipt", {"id": "R-88"}),
step(3, "submit_expense", SUBMIT, final=True, obs=FINAL),
],
}
def calls(run):
return [c for s in run for c in s["tool_calls"]]
def ordered_sequence(run):
"""1.0 when the tool names equal the expected sequence exactly."""
return 1.0 if [c["name"] for c in calls(run)] == EXPECTED_SEQUENCE else 0.0
def unexpected_tools(run):
"""Count of calls to tools outside the expected sequence."""
return sum(c["name"] not in EXPECTED_SEQUENCE for c in calls(run))
def own_step_efficiency(run):
"""expected steps divided by actual steps, capped at 1.0."""
return min(1.0, len(EXPECTED_SEQUENCE) / len(run))
def recovery(run):
"""None when no call failed. Else 1.0 if every failed call was followed by a
successful call to the same tool and the run ended with a final step."""
cs = calls(run)
failed = [i for i, c in enumerate(cs) if not c["success"]]
if not failed:
return None
fixed = all(any(d["name"] == cs[i]["name"] and d["success"] for d in cs[i + 1:])
for i in failed)
return 1.0 if fixed and run[-1]["is_final"] else 0.0
def submit_args_score(run):
"""function_call_accuracy on the submit_expense call (name, parameters, values)."""
sub = [c for c in calls(run) if c["name"] == "submit_expense"]
return call_score(sub[0])
def call_score(call):
got = {"name": call["name"], "arguments": call["arguments"]}
r = evaluate("function_call_accuracy", response=json.dumps(got),
expected_response=json.dumps(EXPECTED_SUBMIT))
return r.score
def kit_scores(run):
out = {}
for m in ["task_completion", "step_efficiency", "tool_selection_accuracy", "trajectory_score"]:
r = evaluate(m, trajectory=run, task=TASK, available_tools=AVAILABLE,
final_result=FINAL, expected_result=FINAL)
assert r.status == "completed", (m, r.status, r.error)
out[m] = r.score
return out
def fmt(x, nd=4):
return "n/a" if x is None else f"{x:.{nd}f}"
def main():
print("SYNTHETIC FIXTURE: three hand-written trajectories, one expense task")
cols = ["task_completion", "step_efficiency", "tool_selection_accuracy", "trajectory_score",
"function_call_accuracy", "ordered_sequence", "unexpected_tools",
"own_step_efficiency", "recovery"]
rows = {}
for name, run in RUNS.items():
k = kit_scores(run)
rows[name] = {**k,
"function_call_accuracy": submit_args_score(run),
"ordered_sequence": ordered_sequence(run),
"unexpected_tools": unexpected_tools(run),
"own_step_efficiency": own_step_efficiency(run),
"recovery": recovery(run)}
w = max(len(c) for c in cols)
header = f"{'metric':<{w}} " + " ".join(f"{n:>21}" for n in RUNS)
print(header)
print("-" * len(header))
for c in cols:
cells = []
for n in RUNS:
v = rows[n][c]
cells.append(f"{(str(v) if c == 'unexpected_tools' else fmt(v)):>21}")
print(f"{c:<{w}} " + " ".join(cells))
print()
print("per task set mean over the 3 runs")
for c in cols:
vals = [rows[n][c] for n in RUNS if rows[n][c] is not None]
print(f"{c:<{w}} {sum(vals) / len(vals):.4f} (n={len(vals)})")
# the point of the fixture: the kit's tool_selection_accuracy does not see the
# extra valid tool, the plain checks do
assert rows["wrong_tool"]["tool_selection_accuracy"] == 1.0
assert rows["wrong_tool"]["unexpected_tools"] == 1
assert rows["wrong_tool"]["ordered_sequence"] == 0.0
assert rows["failed_then_recovered"]["recovery"] == 1.0
assert rows["clean"]["recovery"] is None
# format drift: the amount sent as the string "42.50" instead of the number 42.5
drift = call_score({"name": "submit_expense",
"arguments": {**SUBMIT, "amount": "42.50"}})
print()
print(f"function_call_accuracy with amount \"42.50\" (string) vs 42.5: {fmt(drift)}")
assert drift < 1.0
print()
print("fixture expectations hold")
if __name__ == "__main__":
main()
SYNTHETIC FIXTURE: three hand-written trajectories, one expense task
metric clean wrong_tool failed_then_recovered
--------------------------------------------------------------------------------------------
task_completion 1.0000 1.0000 1.0000
step_efficiency 1.0000 0.8667 0.6667
tool_selection_accuracy 1.0000 1.0000 0.9000
trajectory_score 1.0000 0.9600 0.8700
function_call_accuracy 1.0000 1.0000 1.0000
ordered_sequence 1.0000 0.0000 0.0000
unexpected_tools 0 1 0
own_step_efficiency 1.0000 0.6667 0.6667
recovery n/a n/a 1.0000
per task set mean over the 3 runs
task_completion 1.0000 (n=3)
step_efficiency 0.8445 (n=3)
tool_selection_accuracy 0.9667 (n=3)
trajectory_score 0.9433 (n=3)
function_call_accuracy 1.0000 (n=3)
ordered_sequence 0.3333 (n=3)
unexpected_tools 0.3333 (n=3)
own_step_efficiency 0.7778 (n=3)
recovery 1.0000 (n=1)
function_call_accuracy with amount "42.50" (string) vs 42.5: 0.9000
fixture expectations hold
Aggregating agent scores per run and per task set
Each metric stays its own number per run, then gets averaged per task set, leaving out runs where it does not apply. Across the three runs, recovery averages 1.0 over one run, not three. Repeat each task several times as well, because a single pass hides flaky agents.
Keep a vector per run, one number per metric, instead of one blended score. Our task-set means were 0.8445 for step_efficiency, 0.9667 for tool_selection_accuracy and 0.3333 for ordered_sequence. Recovery’s 1.0000 rests on n=1, since two runs had no failure, so print n beside each mean, as the script does.
The tau-bench authors define pass^k as the chance that all k independent trials of a task succeed, averaged across tasks. If a task runs n times and c runs succeed, the estimate is C(c,k) divided by C(n,k). For any task with a failed trial, that estimate falls toward zero as k grows.
The original repo now points to tau2-bench for the latest version.
Latency targets and the SLOs built on these numbers are a separate job, covered in our post on AI agent reliability metrics.
A minimal metric set by agent type
Build the set from pairs. Each metric sits next to the check that covers its blind spot. Our run showed three: tool selection with the unexpected-tool count (1.0 and 1 on the wrong-tool run), step efficiency with recovery (0.6667 and 1.0 on the retry run), and task completion with one trajectory metric.
Then pick pairs by the failure your agent faces. One lookup and one write risk wrong argument types, so pair argument correctness with a type check on values. Workflows over many tools add or drop calls, so use tool selection plus the unexpected-tool count. If the agent writes a plan, add plan adherence, excluded when no plan step exists.
Timeouts call for step efficiency plus recovery, with fallbacks listed per tool. With several agents, score routing accuracy plus context carried, then child task success, because a dropped handoff starts with the sender. Add a metric only when a run fails in a way the current set cannot name.
Running AI agent evaluation metrics on traced spans in Future AGI
Future AGI runs evals on the spans traceAI records. In Observe you attach preset or custom evals to a project, pick spans, traces or sessions, and run them on historic traffic or continuously on new spans. Scores land as a new column in the trace table, and each span shows its own score.
The docs on evals in Observe describe two run types. The Historical type runs over existing spans that match your filters, then completes. The Continuous type scores new spans as they arrive and stays active, so the checks from a fixture keep scoring production traffic.
When you set up an eval task, you pick Spans, Traces or Sessions as the level it scores. Variable mapping ties each eval input to a span attribute path. Paths use dot notation, with numeric positions such as .0 for array items.
The eval types page covers Code Eval, deterministic Python or JavaScript that calls no model. It also covers LLM-as-Judge, which applies a template’s criteria in one pass. Use Code Eval for unexpected tools and recovery, and LLM-as-Judge for plan quality and context carried.
The built-in trajectory match eval is a code-based check that compares an actual and an expected action list, each a JSON array of names. It runs in strict, unordered, subset or superset mode. Unordered mode treats both lists as sets and scores their Jaccard overlap.
For local runs, agent-learning-kit on GitHub ships tool_selection_accuracy, trajectory_score and function_call_accuracy among its 20+ local deterministic metrics. You call them through from fi.evals import evaluate with no API key. That is how the fixture here was scored.
From the fixture to production spans
The plain checks the script ran, unexpected tools, recovery and our own step efficiency, become Code Eval custom evals. For order, trajectory_match in strict mode scores the length of the matching prefix against the longer list. That gives partial credit where our all-or-nothing check printed 0.0.
Superset mode lowers the score when a tool outside the expected list appears. A retry of the same tool leaves it unchanged. That makes it the built-in counterpart of the unexpected-tool count. In superset mode, flag_for_review would cost the wrong-tool run, and the second lookup would not.
Catching the wrong tool before a receipt reaches review
One extra tool call put the receipt in a reviewer’s queue, and the run still scored 1.0 on its answer. An unexpected-tool count of 1 names that call; an all-or-nothing sequence check flags the run but also fails a correct retry. Pick the metrics your agent type needs and score every run.
On receipt R-88, task completion was right about the end state, and the unexpected-tool count put flag_for_review on the record. On the other run, recovery and step efficiency showed what the retry cost. Each number pointed at a different step, so keep them separate.
Start with the pairs for your agent type. Record a handful of runs per task, and score each run with the same checks. Once the numbers make sense on recorded runs, attach the checks to traced spans, and the next stray review call shows up the day it happens.
Frequently Asked Questions
What are common AI agent evaluation metrics?
What are evals in agentic AI?
What are the types of evals?
What are multi agent evaluation metrics?
What is an agent evaluation trajectory?
Track AI agent token usage per LLM call, then roll it up by step, tool call and run with OpenTelemetry attributes, a dated price table and per-run budgets.
Five use cases for AI agents in banking, the values each must get right, identity checks before any action, and a pre-launch test plan with a runnable check.
Chatbot analytics that finds what failed: sample transcripts by intent, measure resolution and drop-off, score each chat with an LLM judge, rank fixes.