Articles

AI Agent Token Usage Tracking Per Step, Tool Call and Run

Track AI agent token usage per LLM call, then roll it up by step, tool call and run with OpenTelemetry attributes, a dated price table and per-run budgets.

· 14 min read
ai-agent-token-usage agent-token-consumption llm-token-usage opentelemetry llm-cost-tracking traceai python
A semicircular token meter whose needle reads 50,732 tokens for run 1, past a 40,000 token budget mark, with the running totals marked along the dial.
Table of Contents

Your incident triage agent answers on-call alerts with two tools, search_logs and fetch_runbook, and its spend kept rising over the past month. The provider’s usage page splits that spend by model and by day. You still can’t tell whether the log search, a retry or the long system prompt pushed it up.

The bill counts tokens per request, while your agent spends them per step, tool call and run. You need AI agent token usage at those three levels, which is what token usage tracking gives you. A monthly total hides the step that drags a huge log dump through every later call.

Below, you record each LLM call with OpenTelemetry attributes, roll the numbers up the trace without counting anything twice, price them with a dated table and rank the steps that drive cost. The demo agent is tested end to end, and its token counts are synthetic.

TL;DR

  • AI agent token usage piles up in context the loop resends on every call: the system prompt, tool schemas, earlier tool output and retries.
  • Record gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every LLM span, and check the names, since the GenAI conventions are still in Development.
  • Sum only the LLM spans at the bottom of the trace, grouped by step, tool call and run; adding an agent span that repeats its children’s totals doubles the count.
  • Price tokens with a dated table that splits cached from uncached input; cached input costs 10% to 50% of input on the three OpenAI models shown.
  • Rank steps by cost, find the tool output the top step carried, and check a per-run token budget inside the loop with a threshold alert behind it.

Where AI agent token usage goes in a multi-step run

Most AI agent token usage is input that the loop sends again on every call. The system prompt and tool schemas go out with every turn, tool output joins the history and stays there, retries resend the whole context, and sub-agents run loops of their own. A prompt cache cuts the price of that repeated part.

In a 2026 study of agent token costs, eight frontier models on SWE-bench Verified used about 1,000x more tokens on agentic coding tasks than on code reasoning and code chat. Input drove the cost, and runs of one task differed by up to 30x.

Our demo shows the same growth. In run-1, the first call uses 1,078 tokens and the step 3 call uses 16,610, because each call carries everything before it. No call returns more than 61 output tokens. The counts are synthetic, so only the growth carries over.

Coding agents get the proxy-side view in our guide to watching Claude Code token usage through a gateway.

The system prompt and tool schemas resent on every turn

The demo’s system prompt lists 20 triage rules twice as padding, 40 numbered lines, and the JSON schemas for search_logs and fetch_runbook travel with it on every LLM call. That block never changes between calls. It forms a stable prefix ahead of the alert text, which is the part a prompt cache can serve.

Tool output that stays in context

Step 1 calls search_logs, which returns 400 lines of payment gateway timeouts from checkout-api. The agent appends that text to its messages list as a tool message, and nothing removes it. In run-1, three later calls each pay for the same log lines again.

Retries and sub-agent loops

In step 2 the model returns fetch_runbook arguments as truncated JSON. The loop can’t parse them, so it resends the full context plus a short correction, and the retry carries 16,486 input tokens against 16,462 on the first try. A sub-agent runs its own loop one level down, and those calls belong to the parent run.

Cached input versus uncached input

A cached token still counts as an input token, billed at a lower rate. In the demo’s run-1 step 3 call, 16,384 of 16,549 input tokens came from the cache, so your rollup has to keep the two parts apart to get dollars right.

The OpenTelemetry GenAI attributes that record token counts

OpenTelemetry records LLM token usage with two span attributes, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Both belong to the GenAI semantic conventions, which have Development status. Core release v1.45.0 marks every gen_ai attribute as moved to a separate GenAI repository, so check the names again before you build dashboards on them.

The names now live in the OpenTelemetry GenAI semantic conventions repository, with no tagged release as of 2026-11-12, and the v1.45.0 GenAI attribute registry of 2026-11-02 marks the old rows as moved.

gen_ai.usage.input_tokens and gen_ai.usage.output_tokens

The input attribute counts the prompt side of one call and SHOULD include cached tokens and every modality, so a cache hit leaves it unchanged. The output attribute counts the reply and SHOULD include reasoning tokens. Both are Recommended on the inference span, and an invoke_agent span can carry them too.

The older prompt_tokens and completion_tokens names are deprecated. A total_tokens attribute is not in the OTel registry. traceAI, the instrumentation this post uses, writes it as a convenience total. Check before you sum it alongside input and output.

Cache and reasoning attributes after the rename

Two attributes split cached tokens out: cache_read.input_tokens for tokens served from the cache and cache_write.input_tokens for tokens written to it, both under gen_ai.usage. Each SHOULD be included in the input count, so together they describe a share of input. Neither adds tokens on top.

The write attribute used to be called cache_creation, renamed in the GenAI repository ahead of any release. Reasoning gets gen_ai.usage.reasoning.output_tokens, a subset of output, and token metrics under gen_ai.client.inference.usage are in Development too. An instrumentation may write its own cache name. Read one real span first.

Rolling AI agent token usage up to step, tool call and run

AI agent token usage rolls up from the LLM spans at the bottom of the trace, and each span is grouped by the step, tool call and run above it. Count a span only when no span below it carries usage. Agent spans can repeat their children’s totals, and summing every span then doubles the result.

The run is an agent span named incident-triage, with step spans step-1 to step-3 under it. Under each step sit the LLM spans traceAI writes, and steps 1 and 2 hold the tool spans for search_logs and fetch_runbook. Usage lives on the LLM spans, and the demo also sets it on the run span.

Span tree of one demo run, with the incident-triage agent span over three step spans that hold LLM spans with token counts, and tool spans for search_logs and fetch_runbook under the first two steps.

The demo does that on purpose, the way the GenAI conventions let an invoke_agent span carry usage next to its child inference spans. In synthetic run-1, adding every span with usage gives 101,464 tokens and the leaf rollup gives 50,732, a 2.00x double count that the leaf rule removes.

The aggregated token usage issue about summing those numbers is still open upstream.

Below is agent_loop.py, tested end to end. run_agent registers traceAI with an in-memory exporter, instruments the OpenAI client before creating it and wraps each step in a CHAIN span. The client talks to an httpx MockTransport, so it needs no key.

"""Incident triage agent: 2 tools, one retry, one oversized tool output.

The OpenAI Python client talks to an httpx MockTransport, so no provider key and no
network are needed. The usage numbers in each canned response are SYNTHETIC: they are
estimated from request size (characters divided by 4) with a made-up cache rule, so
that they grow the way real usage grows. They are not provider-billed counts.
Spans come from traceAI's OpenAI instrumentor and are kept in an in-memory exporter.
"""
import json
import math

import httpx
import openai
from fi_instrumentation import FITracer, register
from opentelemetry import trace as trace_api
from fi_instrumentation.fi_types import FiSpanKindValues, ProjectType
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from traceai_openai import OpenAIInstrumentor

MODEL = "gpt-4.1"

TRIAGE_RULES = [
    "Start from the alert text and name the service that owns the failing endpoint.",
    "Pull logs for the last 30 minutes before forming any theory about the cause.",
    "Separate errors that started inside the window from errors that were already present.",
    "Check whether the error rate rose with traffic or stayed flat while traffic changed.",
    "Look for a deploy, a config change or a dependency incident near the first error.",
    "Prefer the runbook that matches the failing endpoint over a general runbook.",
    "Quote the exact log line that supports each claim you make in the final answer.",
    "Never guess a customer count. Say that the count is unknown when the logs lack it.",
    "List the next two checks an on-call engineer should run, in order of effort.",
    "Keep the final answer under 120 words and put the likely cause in the first sentence.",
    "If two services fail together, say which one failed first and how you know.",
    "Treat timeouts to a payment provider as a dependency problem until logs show otherwise.",
    "Treat a burst of 5xx right after a deploy as a regression until logs show otherwise.",
    "Do not repeat tool output back to the engineer. Summarize it in your own words.",
    "State which tool calls you made and which fields of their output you relied on.",
    "When a runbook step needs production access, name the step and stop there.",
    "Report times in UTC and keep the same format the log lines use.",
    "If the logs are empty, say so and ask for a wider time window instead of guessing.",
    "Flag any log line that contains a secret or a token and tell the engineer to rotate it.",
    "End with one sentence on what would change your mind about the likely cause.",
] * 2  # synthetic padding so the shared prefix is long enough for the cache rule below

SYSTEM_PROMPT = "You are an incident triage agent for an online store.\n" + "\n".join(
    f"{i + 1}. {r}" for i, r in enumerate(TRIAGE_RULES)
)

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search_logs",
            "description": "Return recent log lines for a service as plain text.",
            "parameters": {
                "type": "object",
                "properties": {
                    "service": {"type": "string"},
                    "minutes": {"type": "integer"},
                },
                "required": ["service", "minutes"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "fetch_runbook",
            "description": "Return the runbook text for a named failure pattern.",
            "parameters": {
                "type": "object",
                "properties": {"name": {"type": "string"}},
                "required": ["name"],
            },
        },
    },
]


def est_tokens(obj) -> int:
    # synthetic estimate: 4 characters per token
    return math.ceil(len(json.dumps(obj, sort_keys=True)) / 4)


def make_handler():
    state = {"prev_prefix": "", "bad_sent": False}

    def handler(request: httpx.Request) -> httpx.Response:
        body = json.loads(request.content)
        msgs = body["messages"]
        n_tool = sum(1 for m in msgs if m["role"] == "tool")
        last = msgs[-1]
        correction = last["role"] == "user" and "not parseable JSON" in last["content"]

        if n_tool == 0:
            call = ("search_logs", json.dumps({"service": "checkout-api", "minutes": 30}))
            message = {"role": "assistant", "content": None}
        elif n_tool == 1 and not state["bad_sent"] and not correction:
            state["bad_sent"] = True
            call = ("fetch_runbook", '{"name": "checkout-5xx"')  # truncated JSON on purpose
            message = {"role": "assistant", "content": None}
        elif n_tool == 1:
            call = ("fetch_runbook", json.dumps({"name": "checkout-5xx"}))
            message = {"role": "assistant", "content": None}
        else:
            call = None
            message = {
                "role": "assistant",
                "content": "Likely cause: payment provider timeouts started at 09:14 UTC "
                "and checkout-api returns 502 when the call exceeds 8 s. Next: check the "
                "provider status page, then raise the timeout to 12 s as the runbook says.",
            }
        if call:
            message["tool_calls"] = [
                {
                    "id": f"call_{n_tool}_{call[0]}",
                    "type": "function",
                    "function": {"name": call[0], "arguments": call[1]},
                }
            ]

        # synthetic usage: prompt size from request body, cache from the shared prefix
        prefix = json.dumps(body["tools"]) + json.dumps(msgs)  # tools first, as a provider prefix would be
        prompt_tokens = math.ceil(len(prefix) / 4)
        shared = 0
        for a, b in zip(prefix, state["prev_prefix"]):
            if a != b:
                break
            shared += 1
        shared_tokens = shared // 4
        cached = (shared_tokens // 128) * 128 if shared_tokens >= 1024 else 0
        state["prev_prefix"] = prefix
        completion_tokens = est_tokens(message)

        return httpx.Response(
            200,
            json={
                "id": "chatcmpl-synthetic",
                "object": "chat.completion",
                "created": 1762900000,
                "model": MODEL,
                "choices": [
                    {
                        "index": 0,
                        "message": message,
                        "finish_reason": "tool_calls" if call else "stop",
                    }
                ],
                "usage": {
                    "prompt_tokens": prompt_tokens,
                    "completion_tokens": completion_tokens,
                    "total_tokens": prompt_tokens + completion_tokens,
                    "prompt_tokens_details": {"cached_tokens": cached},
                    "completion_tokens_details": {"reasoning_tokens": 0},
                },
            },
        )

    return handler


def synthetic_log_dump() -> str:
    # 400 deterministic lines, about 40 KB: the oversized tool output
    lines = []
    for i in range(400):
        sec = i * 4
        lines.append(
            f"2026-11-12T09:{14 + sec // 3600:02d}:{(sec // 60) % 60:02d}Z checkout-api "
            f"{'ERROR' if i % 3 else 'WARN'} upstream payment-gateway timeout after 8000ms "
            f"request_id=req-{i:05d} route=/v2/checkout/confirm status=502 retry={i % 2}"
        )
    return "\n".join(lines)


def run_agent(questions):
    """Run the loop once per question and return all finished spans."""
    exporter = InMemorySpanExporter()
    provider = register(
        project_name="token-usage-demo",
        project_type=ProjectType.OBSERVE,
        batch=False,
        verbose=False,
        span_exporter=exporter,
    )
    OpenAIInstrumentor().instrument(tracer_provider=provider)
    client = openai.OpenAI(  # created after instrument()
        api_key="mock-key-no-provider",
        http_client=httpx.Client(transport=httpx.MockTransport(make_handler())),
        max_retries=0,
    )
    fi_tracer = FITracer(provider.get_tracer("triage-agent"))

    @fi_tracer.tool(name="search_logs", description="Recent log lines for a service")
    def search_logs(service: str, minutes: int) -> str:
        return synthetic_log_dump()

    @fi_tracer.tool(name="fetch_runbook", description="Runbook text for a failure pattern")
    def fetch_runbook(name: str) -> str:
        return "Runbook checkout-5xx: confirm provider status, then raise the client timeout to 12 s."

    tools = {"search_logs": search_logs, "fetch_runbook": fetch_runbook}

    @fi_tracer.agent(name="incident-triage")
    def triage(question: str) -> str:
        messages = [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ]
        answer = ""
        seen = {"in": 0, "out": 0}
        for step in range(1, 4):
            with fi_tracer.start_as_current_span(
                f"step-{step}", fi_span_kind=FiSpanKindValues.CHAIN
            ) as step_span:
                step_span.set_attribute("agent.step.index", step)
                extra = []  # correction message only lives for the retry
                for attempt in (1, 2):
                    resp = client.chat.completions.create(
                        model=MODEL, messages=messages + extra, tools=TOOLS
                    )
                    seen["in"] += resp.usage.prompt_tokens
                    seen["out"] += resp.usage.completion_tokens
                    msg = resp.choices[0].message
                    if not msg.tool_calls:
                        answer = msg.content
                        break
                    tc = msg.tool_calls[0]
                    try:
                        args = json.loads(tc.function.arguments)
                    except json.JSONDecodeError:
                        extra = [
                            {"role": "user", "content": "The tool arguments were not parseable JSON. Call the tool again."}
                        ]
                        continue
                    messages.append(
                        {
                            "role": "assistant",
                            "content": None,
                            "tool_calls": [
                                {
                                    "id": tc.id,
                                    "type": "function",
                                    "function": {"name": tc.function.name, "arguments": tc.function.arguments},
                                }
                            ],
                        }
                    )
                    out = tools[tc.function.name](**args)
                    messages.append({"role": "tool", "tool_call_id": tc.id, "content": out})
                    break
            if answer:
                break
        # The agent span repeats the sum of its LLM calls, the way an OTel GenAI invoke_agent
        # span may carry usage next to its child spans. Adding these makes parent and child
        # both hold the same tokens, which is what the rollup has to handle.
        run_span = trace_api.get_current_span()
        run_span.set_attribute("gen_ai.usage.input_tokens", seen["in"])
        run_span.set_attribute("gen_ai.usage.output_tokens", seen["out"])
        return answer

    answers = [triage(q) for q in questions]
    provider.force_flush()
    return answers, exporter.get_finished_spans()

Setup for other frameworks and exporters is in how traceAI instruments LLM calls with OpenTelemetry.

The leaf rule that stops double counting

In rollup.py below, build counts a span only if it carries usage and no span below it does. That picks the seven LLM spans, and the script asserts they equal the spans traceAI marks LLM. An open issue in the same repository adds that summed usage breaks under sampling, so set budgets from unsampled traces.

"""Roll LLM-span token usage up to step, tool call and run, then price it.

Run: python rollup.py
Usage numbers are SYNTHETIC (see agent_loop.py): request-size estimates, not provider counts.
Prices are a dated copy of the OpenAI standard price list, short context.
"""
from collections import defaultdict
from decimal import Decimal as D

from agent_loop import run_agent

# USD per 1M tokens: (input, output, cached input). Standard processing, short context.
# Source: https://developers.openai.com/api/docs/pricing (page time 2026-11-12 03:30 GMT)
PRICE_DATE = "2026-11-12"
PRICES = {
    "gpt-4.1": (D("2.00"), D("8.00"), D("0.50")),
    "gpt-4o-mini": (D("0.15"), D("0.60"), D("0.075")),
    "gpt-5.4": (D("2.50"), D("15.00"), D("0.25")),
}
# Policy choice for this demo, not a provider limit.
RUN_TOKEN_BUDGET = 40_000

IN, OUT, TOTAL = "gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens", "gen_ai.usage.total_tokens"
# traceAI writes the first name; the OTel GenAI draft uses the second. Read whichever the span has.
CACHE_READ_NAMES = ("gen_ai.usage.input_tokens.cache_read", "gen_ai.usage.cache_read.input_tokens")
MILLION = D(1_000_000)


def usage_of(span):
    a = span.attributes
    if IN not in a and OUT not in a:
        return None
    cached = next((a[n] for n in CACHE_READ_NAMES if n in a), 0)
    return {"in": a.get(IN, 0), "out": a.get(OUT, 0), "cached": cached, "total": a.get(TOTAL)}


def money(model, u, use_cache=True):
    p_in, p_out, p_cached = PRICES[model]
    cached = u["cached"] if use_cache else 0
    return ((u["in"] - cached) * p_in + cached * p_cached + u["out"] * p_out) / MILLION


def fmt(x):
    return f"${x:.5f}"


def build(spans):
    by_id = {s.context.span_id: s for s in spans}
    kids = defaultdict(list)
    for s in spans:
        if s.parent is not None:
            kids[s.parent.span_id].append(s)

    def has_usage_below(s):
        return any(usage_of(c) or has_usage_below(c) for c in kids[s.context.span_id])

    # Count usage once: only spans that carry usage and have no usage-carrying span below them.
    leaves = [s for s in spans if usage_of(s) and not has_usage_below(s)]
    return by_id, leaves


def step_of(span, by_id):
    while span is not None:
        if "agent.step.index" in span.attributes:
            return span.attributes["agent.step.index"]
        span = by_id.get(span.parent.span_id) if span.parent else None
    return None


def main():
    questions = [
        "checkout-api is returning 502 since 09:14 UTC. What is going on?",
        "Is the 502 on checkout-api still happening? Same alert, 09:40 UTC.",
    ]
    _, spans = run_agent(questions)
    spans = sorted(spans, key=lambda s: s.start_time)
    by_id, leaves = build(spans)

    roots = [s for s in spans if s.parent is None]
    run_name = {r.context.trace_id: f"run-{i + 1}" for i, r in enumerate(roots)}
    for s in leaves:  # the leaf rule and traceAI's LLM kind must pick the same spans
        assert s.attributes["gen_ai.span.kind"] == "LLM"
        u = usage_of(s)
        assert u["total"] == u["in"] + u["out"]
    assert len(leaves) == sum(1 for s in spans if s.attributes.get("gen_ai.span.kind") == "LLM")

    print("SYNTHETIC USAGE: token counts are request-size estimates from agent_loop.py")
    print("(4 characters per token, made-up cache rule), not counts billed by a provider.")
    print()
    print(f"PRICE TABLE, USD per 1M tokens, standard processing, short context, dated {PRICE_DATE}")
    print("source: https://developers.openai.com/api/docs/pricing")
    print(f"{'model':<14}{'input':>8}{'output':>8}{'cached':>8}{'cached/input':>14}")
    for m, (i, o, c) in PRICES.items():
        print(f"{m:<14}{i:>8}{o:>8}{c:>8}{c / i:>14.1%}")
    print()

    model = leaves[0].attributes["gen_ai.request.model"]

    # 1. every LLM call
    print("1. LLM CALLS (leaf spans with usage), priced as", model)
    print(f"{'run':<7}{'step':>5}{'call':>5}{'input':>8}{'cached':>8}{'output':>8}{'total':>8}{'cost':>11}")
    call_no = defaultdict(int)
    rows = []
    for s in leaves:
        u = usage_of(s)
        run = run_name[s.context.trace_id]
        step = step_of(s, by_id)
        call_no[(run, step)] += 1
        c = money(model, u)
        rows.append({"span": s, "run": run, "step": step, "n": call_no[(run, step)], "u": u, "cost": c})
        print(f"{run:<7}{step:>5}{call_no[(run, step)]:>5}{u['in']:>8}{u['cached']:>8}{u['out']:>8}{u['in'] + u['out']:>8}{fmt(c):>11}")
    print()

    # 2. double counting
    print("2. DOUBLE COUNTING: sum over every span with usage vs leaf spans only")
    for run in sorted(set(run_name.values())):
        tid = next(t for t, n in run_name.items() if n == run)
        every = sum(usage_of(s)["in"] + usage_of(s)["out"] for s in spans if s.context.trace_id == tid and usage_of(s))
        once = sum(r["u"]["in"] + r["u"]["out"] for r in rows if r["run"] == run)
        agent = next(r for r in roots if r.context.trace_id == tid)
        print(f"{run}: naive sum {every} tokens, leaf rollup {once} tokens, naive is {every / once:.2f}x "
              f"(agent span repeats {usage_of(agent)['in'] + usage_of(agent)['out']})")
    print()

    # 3. per step, ranked by cost
    steps = defaultdict(lambda: {"calls": 0, "in": 0, "cached": 0, "out": 0, "cost": D(0), "retry_cost": D(0), "retry_calls": 0})
    for r in rows:
        t = steps[(r["run"], r["step"])]
        t["calls"] += 1
        t["in"] += r["u"]["in"]
        t["cached"] += r["u"]["cached"]
        t["out"] += r["u"]["out"]
        t["cost"] += r["cost"]
        if r["n"] > 1:
            t["retry_calls"] += 1
            t["retry_cost"] += r["cost"]
    run_cost = defaultdict(D)
    run_tokens = defaultdict(int)
    for (run, _), t in steps.items():
        run_cost[run] += t["cost"]
        run_tokens[run] += t["in"] + t["out"]
    print("3. PER STEP, ranked by cost (retry calls = calls after the first inside a step)")
    print(f"{'rank':<5}{'run':<7}{'step':>5}{'calls':>6}{'retries':>8}{'input':>8}{'cached':>8}{'output':>8}{'cost':>11}{'of run':>8}")
    ranked = sorted(steps.items(), key=lambda kv: kv[1]["cost"], reverse=True)
    for i, ((run, step), t) in enumerate(ranked, 1):
        print(f"{i:<5}{run:<7}{step:>5}{t['calls']:>6}{t['retry_calls']:>8}{t['in']:>8}{t['cached']:>8}{t['out']:>8}{fmt(t['cost']):>11}{t['cost'] / run_cost[run]:>8.1%}")
    print()

    # 4. per tool call: estimate net context growth and its carry cost, not exact tool billing
    print("4. PER TOOL CALL: estimated net context growth on the next LLM call and estimated carry cost")
    print("(estimated carry cost allocates money already counted in table 3, do not add the tables)")
    print(f"{'run':<7}{'tool':<15}{'net growth':>12}{'calls':>8}{'estimated carry cost':>22}")
    tool_rows = []
    for ts in (s for s in spans if s.attributes.get("gen_ai.span.kind") == "TOOL"):
        tid = ts.context.trace_id
        step = step_of(ts, by_id)
        in_trace = [r for r in rows if r["span"].context.trace_id == tid]
        asked = [r for r in in_trace if r["step"] == step and r["span"].end_time <= ts.start_time][-1]
        later = [r for r in in_trace if r["span"].start_time >= ts.end_time]
        growth = later[0]["u"]["in"] - asked["u"]["in"] if later else 0
        carry = sum(
            growth * (money(model, {**r["u"], "out": 0}) / r["u"]["in"]) for r in later
        )
        tool_rows.append((run_name[tid], ts.name, growth, len(later), carry))
    for run, name, growth, n, carry in sorted(tool_rows, key=lambda x: x[4], reverse=True):
        print(f"{run:<7}{name:<15}{growth:>12}{n:>8}{fmt(carry):>22}")
    print()

    # 5. retries and cache
    retry_cost = sum(t["retry_cost"] for t in steps.values())
    retry_tokens = sum(r["u"]["in"] + r["u"]["out"] for r in rows if r["n"] > 1)
    all_cost = sum(run_cost.values())
    print("5. RETRIES AND CACHE")
    print(f"retry calls: {sum(t['retry_calls'] for t in steps.values())}, {retry_tokens} tokens, {fmt(retry_cost)} of {fmt(all_cost)} ({retry_cost / all_cost:.1%})")
    uncached = sum(money(model, r["u"], use_cache=False) for r in rows)
    print(f"all calls with cached-input rate: {fmt(all_cost)}; same tokens billed as all uncached input: {fmt(uncached)}")
    print(f"cache lowers this total by {1 - all_cost / uncached:.1%} on {model} "
          f"(cached input is {PRICES[model][2] / PRICES[model][0]:.0%} of the input price)")
    print()
    print("same tokens, other models, with and without the cached-input rate")
    print(f"{'model':<14}{'cached rate':>13}{'all uncached':>14}{'cache saves':>13}")
    for m in PRICES:
        with_c = sum(money(m, r["u"]) for r in rows)
        no_c = sum(money(m, r["u"], use_cache=False) for r in rows)
        print(f"{m:<14}{fmt(with_c):>13}{fmt(no_c):>14}{1 - with_c / no_c:>13.1%}")
    print()

    # 6. per-run budget
    print(f"6. PER-RUN TOKEN BUDGET ({RUN_TOKEN_BUDGET} tokens, a policy value chosen for this demo)")
    for run in sorted(run_tokens):
        running = 0
        hit = None
        for r in (r for r in rows if r["run"] == run):
            running += r["u"]["in"] + r["u"]["out"]
            if running > RUN_TOKEN_BUDGET and hit is None:
                hit = (r["step"], r["n"], running)
        if hit:
            print(f"ALERT {run}: {run_tokens[run]} tokens, over budget at step {hit[0]} call {hit[1]} (running total {hit[2]})")
        else:
            print(f"ok    {run}: {run_tokens[run]} tokens, under budget, cost {fmt(run_cost[run])}")


if __name__ == "__main__":
    main()

Grouping by step, tool call and run

step_of walks parents up to the span with agent.step.index, and the run is the trace itself. The input difference before and after a tool call estimates net context growth, not the tool output’s exact token count. It includes message envelopes and other context changes, such as removing the retry correction. Section 1 below holds the rows behind every later table, and section 2 checks the double count.

1. LLM CALLS (leaf spans with usage), priced as gpt-4.1
run     step call   input  cached  output   total       cost
run-1      1    1    1026       0      52    1078   $0.00247
run-1      2    1   16462    1024      48   16510   $0.03177
run-1      2    2   16486   16384      48   16534   $0.00878
run-1      3    1   16549   16384      61   16610   $0.00901
run-2      1    1    1027       0      52    1079   $0.00247
run-2      2    1   16462    1024      48   16510   $0.03177
run-2      3    1   16550   16384      61   16611   $0.00901

2. DOUBLE COUNTING: sum over every span with usage vs leaf spans only
run-1: naive sum 101464 tokens, leaf rollup 50732 tokens, naive is 2.00x (agent span repeats 50732)
run-2: naive sum 68400 tokens, leaf rollup 34200 tokens, naive is 2.00x (agent span repeats 34200)

Rollup diagram where the seven LLM call counts sum into step totals and then into the run-1 and run-2 totals of 50,732 and 34,200 tokens.

Turning agent token consumption into dollars with a dated price table

Agent token consumption turns into dollars one call at a time. Multiply uncached input by the input price, cached input by the cached price and output by the output price, then add. Date the table you use, because prices change, and cached input on these three OpenAI models costs 10% to 50% of normal input.

OpenAI counts cached tokens inside the input count, so uncached input is input minus cached. Chat Completions reports prompt_tokens with the cached count in prompt_tokens_details.cached_tokens, while the Responses API uses input_tokens and input_tokens_details.cached_tokens. Table 1 takes three models from OpenAI API pricing.

ModelInputOutputCached inputCached as share of input
gpt-4.12.008.000.5025.0%
gpt-4o-mini0.150.600.07550.0%
gpt-5.42.5015.000.2510.0%

Table 1 is USD per 1M tokens, standard processing, short context, as of 2026-11-12; batch and other tiers differ. Priced on gpt-4.1, the seven demo calls cost $0.09528 with the cached rate and $0.17208 billed as all uncached input, so the cached rate cuts the total by 44.6%.

Per-user and per-prompt attribution is covered in our LLM cost tracking practices.

Why cached input changes the math

Only the cached rate moves between the two pricing passes. The same tokens save 29.8% on gpt-4o-mini and 53.1% on gpt-5.4, because cached input costs 50% and 10% of input on those models against 25% on gpt-4.1. One flat discount would misprice two of the three, so keep the rate per model.

OpenAI and Anthropic count cached input differently

Anthropic’s usage.input_tokens counts only tokens after the last cache breakpoint, the reverse of OpenAI, so its total input is cache reads plus cache writes plus input_tokens. Normalize both providers to the OTel rule, where input includes cached tokens, before you sum them.

The OpenAI prompt caching guide and Anthropic prompt caching docs define these fields.

As of 2026-11-12, Anthropic pricing bills cache hits at 0.05x base input on Claude Opus 5.5 and Sonnet 5.5, 0.025x on the Fable and Mythos 5.1 and 5.2 models, and 0.1x on the others. Keep those multipliers in the same dated table.

Some OpenAI models add a write rate. On GPT-5.6 and later, written tokens cost 1.25x input for the default 30-minute lifetime and 2.5x for the 12-hour one in the Responses API, so subtract cache_write_tokens and price them at that rate. Table 1’s models have no write rate.

Finding the steps that drive cost in an agent run

Rank every step by its share of the run’s cost, then look at what the top step carried into context. In the synthetic demo, one step that carried an oversized log dump and a retry took 77.9% of its run’s cost, and the log output stayed in context for each call after it.

RankRunStepCallsRetriesInputCachedOutputCostShare of run
1run-122132,94817,40896$0.0405577.9%
2run-221016,4621,02448$0.0317773.5%
3run-231016,55016,38461$0.0090120.8%
4run-131016,54916,38461$0.0090117.3%
5run-21101,027052$0.002475.7%
6run-11101,026052$0.002474.7%

Table 2 ranks both runs’ steps by gpt-4.1 cost on synthetic usage. Run-1 step 2 leads because it made the first call carrying the log output, and its retry sent that context again. Run-2 step 2 had no retry and still took 73.5%.

Our guide to cutting agent cost once you know the driver covers the order of fixes.

The oversized tool output behind the top step

The per-tool view estimates net context growth and carry cost. In run-1, search_logs was followed by 15,436 tokens of net context growth, carried across 3 calls for an estimated $0.04524. After fetch_runbook, net growth was 63 tokens, versus 88 in run-2 for the same output. The carry estimate applies each later call’s blended cached/uncached input rate, not the tool output’s actual cache allocation. That estimated cost already sits inside Table 2. Never add it to the step totals.

What a retry costs

The retry in run-1 step 2 was one call of 16,534 tokens, $0.00878 of the $0.09528 both runs cost, or 9.2%. It stayed cheap because 16,384 of its 16,486 input tokens hit the cache, which a model without a cached rate bills in full.

Per-run token budgets and alerts inside the agent loop

A per-run token budget works when the loop checks the running total after every LLM call, where the code can still stop the run or hand it to a person. Put a threshold alert on token usage over time behind it to catch slow drift that no single run crosses.

The demo replays that check over the recorded spans with a 40,000-token budget, a policy value chosen for the demo. Run-1’s running total goes 1,078, 17,588, 34,122, then 50,732, crossing at step 3 call 1, and run-2 ends at 34,200.

A post-response check can prevent another call, but the completed call can overshoot the budget: run-1 exceeds it by 10,732 tokens. A hard cap needs a pre-dispatch check that reserves the next call’s input and maximum output against the remaining budget.

6. PER-RUN TOKEN BUDGET (40000 tokens, a policy value chosen for this demo)
ALERT run-1: 50732 tokens, over budget at step 3 call 1 (running total 50732)
ok    run-2: 34200 tokens, under budget, cost $0.04325

The check reads leaf usage, because the agent span reports its total only after the run ends. Sub-agent calls roll into the parent run’s running total, so one budget covers the whole tree, and a gateway can enforce its own limits at a separate layer.

Where the budget check goes in the loop

Put the check after each LLM response and before the next tool call, right after client.chat.completions.create, where the demo already adds prompt and completion tokens to a running count. On a breach, stop the loop, return the partial answer and flag the run for a person to review.

What a token usage tracker should alert on

A token usage tracker should alert on tokens per run over the budget, daily token totals per model, cost per day and the retry share of calls. Each is a time series a threshold alert checks on a schedule, while the budget sits in the loop. A rising retry share flags runs like run-1 step 2.

Tracking agent tokens and cost in Future AGI Observe

Future AGI traceAI is built on OpenTelemetry. It instruments the OpenAI client and writes gen_ai.usage token counts on every LLM span. The Future AGI platform then computes the USD cost on ingest from the model and token counts. In Observe you chart tokens and cost over time and set threshold alerts on token usage.

traceAI-openai records input, output and total token counts on each LLM span, plus a cache-read count, the same spans the rollup read. On ingest the platform computes each LLM span’s USD cost from the model, input tokens and output tokens, and keeps any cost you already set.

Instrumentors for other frameworks are in traceAI on GitHub.

Sending the spans with the traceAI OpenAI instrumentor

To send the spans to Future AGI, call register without the in-memory span_exporter, as future_agi_export.py does by following the traceAI OpenAI integration page.

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from traceai_openai import OpenAIInstrumentor


def instrument_for_observe(project_name: str = "incident-triage"):
    provider = register(project_name=project_name, project_type=ProjectType.OBSERVE)
    OpenAIInstrumentor().instrument(tracer_provider=provider)  # create the OpenAI client after this
    return provider

Set FI_API_KEY and FI_SECRET_KEY, call instrument_for_observe() and then create the OpenAI client, as the traceAI OpenAI page says to initialize instrumentation before creating a client. The loop’s chat.completions.create calls stay as they are, and each now reaches the incident-triage project as an LLM span.

Dashboards and alerts for tokens and cost

In Observe dashboards, add a token usage widget for the triage project, summed per day and grouped by model, then a cost widget with the same settings. Together they show which model’s tokens rose on which day, with cost computed by the platform.

Then set Observe alerts with a threshold on token usage or daily tokens spent, sent to email, Slack or both, so a scheduled check backs up the budget in your loop.

From one daily bill line to a token budget on every agent run

The triage agent’s rising spend comes down to one row: a step, the tool output it carried and a retry. Getting there takes four moves on its LLM calls: instrument them, roll usage up the trace without the double count, price it with a dated table and rank the steps.

Then add the budget check after every LLM response. The next time search_logs returns an oversized dump, the run stops after the call that crosses the budget; that call can overshoot it. A person sees the flag that day, instead of a bigger number on next month’s bill.

Start with one real trace from your own agent. Export its spans, run the leaf rule over them and check whether your agent span repeats its children’s totals. Rank the steps by cost, and the top row names what to trim first.

Frequently Asked Questions

How many tokens does agentic AI use?

Agentic coding tasks used about 1,000 times more tokens than code reasoning or code chat in a 2026 study of eight frontier models on SWE-bench Verified (arXiv 2604.22750). Input tokens drove the cost. The same task also varied up to 30 times between runs, so budget from measured runs.

How do you track AI agent token usage?

Record gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every LLM call span, then sum only those leaf spans by their parent step, tool call and run. An OpenTelemetry instrumentor such as traceAI writes the attributes for you. Skip the agent span's own totals, or the run counts twice.

How much do AI tokens cost?

AI tokens are priced per million, at a rate that differs by model and by token type: uncached input, cached input and output each have their own price. On OpenAI standard pricing as of 12 November 2026, gpt-4.1 costs $2.00 per million input tokens, $0.50 cached and $8.00 output. An agent pays them on every call, retries included.

How to reduce token usage in AI agents?

Measure tokens per step first, then shrink what the costliest step resends, since most agent tokens are input the loop sends again on every call. Trim oversized tool output before it goes back into context. Keep the system prompt and tool schemas identical between calls so the prompt cache hits. Cap retries per step.

Do agents consume tokens?

Yes, every LLM call an agent makes consumes input and output tokens, and the loop resends the system prompt, tool schemas and earlier tool output on each call. Retries and sub-agent loops consume tokens too. Cached input still counts as input tokens, and the provider bills it at a lower rate.
Related Articles
View all