What Is LLM Monitoring?

What Is LLM Monitoring?

LLM monitoring is the continuous production tracking of model and agent behavior after deployment. traces, spans, prompts, completions, tool calls, latency, token usage, cost, safety events, and quality scores. It is the operational arm of LLM observability, the layer that watches signals continuously, thresholds them, and routes alerts. In a FutureAGI workflow it shows up on live traceAI spans and fi.client.Client.log records, where engineers alert on eval drift, p99 latency, rising cost-per-trace, or unsafe responses long before a customer screenshot lands in support.

If you are reading this as a senior engineer in May 2026, the relevant question is not “should I monitor my LLM?”. every serious team monitors. The question is what the 2026 stack looks like now that agent traces routinely span 10–50 spans per request, reasoning models burn 40K output tokens for a single turn, and tool-calling failures sit underneath visibly-passing HTTP 200 responses. This page is an opinionated tour of that 2026 monitoring stack and how FutureAGI implements it.

Why LLM monitoring matters in production LLM and agent systems

The failure mode is quiet degradation. A support agent keeps returning 200s while a retriever serves stale policy text, a tool retries until cost spikes, or a provider weight update changes refusal behavior. Without LLM monitoring, the first alert is a customer screenshot, a compliance review, or a finance report showing token spend doubled overnight after a silent provider rollout.

Developers feel it as non-reproducible bugs. SREs see p99 latency move but cannot tell which prompt, model, route, or tool caused it. Product teams see thumbs-down rates climb without knowing whether the root cause is retrieval, generation, routing, or prompt drift. Compliance teams lose the audit trail for why a regulated answer was produced. In our 2026 evals we routinely see teams who instrumented well but never wired alerts. they have the data, they just discover the regression three weeks late.

Common symptoms show up as uneven trace shapes, rising llm.token_count.prompt, longer time-to-first-token, repeated tool spans, eval-fail-rate-by-cohort, and cost-per-trace outliers. Agentic systems sharpen this because one user request now triggers planning, retrieval, tool selection, tool execution, critique, and a final synthesis. A weak step early in the trajectory poisons every later step while the top-level request still appears successful. Monitoring turns those hidden step failures into operational signals.

The 2026 shift: from APM to AI-native monitoring

Classic APM watches HTTP status, error rate, CPU, and request latency. Those are necessary in 2026 but no longer sufficient. The frontier monitoring problems are agent-specific: tool-selection drift across MCP servers after a tool description rewrite; hallucination rate climbing because a RAG index was reindexed; output-token spend doubling because a reasoning model now thinks longer; refusal rate flipping after a Claude Opus 4.7 or GPT-5.x weight refresh. Datadog, New Relic, and traditional APM do not surface those without a layer above them. FutureAGI’s monitoring stack is built for those problems specifically. span-attached evaluator scores, agent-graph topology, prompt-version tagging, and cohort-sliced quality dashboards. Unlike LangSmith, which treats evals and traces as separate datasets joined by primary key, FutureAGI writes evaluator scores back onto the trace itself so a single dashboard can answer “where did latency, cost, and quality break together?”

How FutureAGI monitors LLM and agent systems

FutureAGI’s approach is to treat monitoring as the production side of evaluation, not a separate dashboard. Instrumentation is one import. A LangChain RAG app calls register(project_name="prod-rag") and LangChainInstrumentor().instrument() and every chain step emits an OpenTelemetry span. Spans carry gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.server.time_to_first_token, gen_ai.cost.total, and fi.span.kind (LLM, RETRIEVER, TOOL, AGENT, CHAIN, GUARDRAIL). The same application can also call fi.client.Client.log for cases that need explicit SDK logging. conversations, tags, custom timestamps.

A practical 2026 workflow: an agent answers a healthcare benefits question, traceAI records spans for retrieval and generation, and llm.token_count.prompt, llm.token_count.completion, gen_ai.server.time_to_first_token, and gen_ai.evaluation.score.value (see the OpenTelemetry GenAI semantic conventions) attach to the trace. FutureAGI samples the answer for Groundedness and HallucinationScore. If groundedness drops below 0.75 for the “benefits-policy” cohort, the dashboard creates an alert and the engineer opens the exact failing trace, sees the retrieved chunks, sees the prompt version, and ships a fix.

Span-attached evals: the differentiator

The piece that separates 2026-grade monitoring from a glorified log viewer is span-attached eval scores. A HallucinationScore, Groundedness, AnswerRelevancy, ToolSelectionAccuracy, or Faithfulness evaluator runs on every sampled span and writes its verdict back as gen_ai.evaluation.score.value with a reason. Filter the dashboard to “spans where citation grounding dropped below 0.7 in the last 24h” and that becomes your review queue. Filter to “spans where ToolSelectionAccuracy < 0.5 after the May 12 deploy” and you have a regression cohort. Unlike a trace-only setup in tools such as LangSmith or Langfuse, where the trace and the eval live in different tables, FutureAGI keeps them on the same span so alerts can combine quality and runtime in one threshold expression.

FutureAGI’s approach is to treat the highest-signal alerts as joint conditions: groundedness plus retriever latency, hallucination score plus prompt version, cost-per-trace plus retry count. In our 2026 evals the single-signal alert is a false-positive factory; the joint alert catches real incidents. The empirical case for online evals is stark: on RAGTruth’s 18K labeled chunks, frontier models still fail groundedness on 5–8% of answers, and HaluEval’s 35K Q&A set shows GPT-4-class models hallucinating at ~16.4%. both well above what any uniform 1% sampling regime will catch without span-attached evaluator scores.

Agent Command Center: monitoring at the gateway

The runtime piece is the Agent Command Center gateway. It sits in front of every model provider, captures every request and response, applies pre- and post-guardrails, enforces a routing policy, and emits the same gen_ai.* semantic attributes traceAI consumes. When OpenAI’s GPT-5.1 turns flaky for an hour, the gateway’s model fallback config moves traffic to Claude Opus 4.7 or Gemini 3 Pro and tags the affected traces; the monitoring dashboard surfaces the failover with a banner instead of a paging storm. The semantic-cache records hit rate per route, the cost-optimized router records savings per request, and traffic-mirroring runs a candidate model in shadow on real traffic. Monitoring without a gateway sees what happened; monitoring with a gateway sees what happened and routes around it.

For example, an internal team running a customer-support agent on GPT-5 and Claude Opus 4.7 in A/B sees Claude’s Groundedness score drop 6 points after a model refresh. The gateway’s routing policy is updated to send 100% of traffic to GPT-5 for the affected cohort, the alert is acknowledged, and a regression eval runs against the candidate before traffic is restored. That entire loop happens inside FutureAGI; the engineer never leaves the platform.

Per-cohort monitoring beats global averages

A global pass rate of 92% can hide a 60% pass rate on refund workflows. FutureAGI’s monitoring slices every metric by tenant, route, model, prompt version, cohort tag, and agent.trajectory.step. A dashboard for an agentic product looks like a matrix. quality and cost on the Y axis, agent role on the X axis. not a single line chart. The teams that ship with confidence in 2026 monitor at cohort granularity because that is where regressions live.

How to measure or detect LLM monitoring health

Treat LLM monitoring itself as a measurable layer. The signals are:

• Trace coverage. percentage of production LLM calls with a complete trace tree (trace id, parent span id, model name, prompt version, route tag). Target 99%+ for critical paths.
• Span completeness. every LLM span has gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, duration, status, and fi.span.kind. Missing fields hide failure modes.
• Token and cost drift. llm.token_count.prompt, llm.token_count.completion, and token-cost-per-trace by route, tenant, and model. A 30% step-change is almost always a real incident.
• Latency. p99 span duration and gen_ai.server.time_to_first_token, split by model and streaming path.
• Quality. Groundedness returns whether an answer is supported by supplied context; HallucinationScore flags unsupported claims; AnswerRelevancy checks the answer addressed the request; Faithfulness and ContextRelevance score RAG pipelines; ToolSelectionAccuracy and TaskCompletion score agents.
• Safety. PromptInjection, ProtectFlash, BiasDetection, Toxicity, and PII detect production attacks and policy violations in real time.
• User proxy. thumbs-down rate, escalation rate, refund requests, and human-review overrides by cohort.
• Drift signals. distribution shifts in input embeddings, output length, refusal rate, and per-cohort fail rate.

A reference snapshot of 2026 monitoring signals

Signal	OTel attribute or evaluator	Why it matters in 2026
Prompt token spend	gen_ai.usage.input_tokens	Long-context prompts (1M tokens on Gemini 3.x) make prompt cost dominant.
Output token spend	gen_ai.usage.output_tokens	Reasoning models (o-series, Claude with extended thinking) burn 10–40K out tokens per turn.
TTFT	gen_ai.server.time_to_first_token	Streaming UX SLA; provider weight refreshes shift TTFT silently.
Cost per trace	gen_ai.cost.total	Routing policy effectiveness only shows up here.
Groundedness	Groundedness	RAG quality after corpus reindex.
Hallucination	HallucinationScore	Provider weight refresh regressions.
Tool selection	ToolSelectionAccuracy	MCP server churn breaks tool routing.
Task completion	TaskCompletion	The only honest end-to-end agent signal.
Trajectory	TrajectoryScore	Per-step agent health across a multi-agent system.
Prompt injection	PromptInjection, ProtectFlash	2026 indirect-injection attacks via MCP resources.
PII leakage	PII	Regulated workloads; pre-storage redaction.
Refusal rate	custom + AnswerRefusal	Refusal cliffs after provider updates.

Minimal pairing snippet:

from fi.evals import Groundedness, HallucinationScore, ToolSelectionAccuracy

groundedness = Groundedness()
halluc = HallucinationScore()
tool_acc = ToolSelectionAccuracy()

for sampled_trace in production_sample:
    g = groundedness.evaluate(
        input=sampled_trace.user_input,
        context=sampled_trace.retrieved,
        output=sampled_trace.answer,
    )
    h = halluc.evaluate(
        input=sampled_trace.user_input,
        output=sampled_trace.answer,
    )
    t = tool_acc.evaluate(
        trajectory=sampled_trace.steps,
        expected_tool=sampled_trace.expected_tool,
    )
    sampled_trace.attach_scores(groundedness=g, hallucination=h, tool=t)

Wiring an online evaluator onto a live traceAI span:

from fi.evals import Groundedness
from traceai.opentelemetry import current_span

groundedness = Groundedness()

def on_rag_answer(user_q, retrieved_chunks, answer):
    span = current_span()
    score = groundedness.evaluate(
        input=user_q,
        context=retrieved_chunks,
        output=answer,
    )
    span.set_attribute("gen_ai.evaluation.score.value", score.value)
    span.set_attribute("gen_ai.evaluation.score.name", "Groundedness")
    span.set_attribute("gen_ai.evaluation.reason", score.reason)
    if score.value < 0.75:
        span.set_attribute("fi.alert.cohort", "rag-grounding-regression")

Alerting thresholds that actually fire correctly

Single-threshold alerts on global averages create noise. The thresholds that earn paging rights in 2026 are joint conditions: Groundedness rolling mean below 0.75 AND retriever latency above 800ms AND cohort = “billing”. That triple-condition fires when something real happens and stays quiet when one signal drifts on its own. Combine quality, runtime, and cohort, then route to the engineer who owns the evaluator or prompt that triggered.

What “good monitoring” looks like in a 2026 production review

A reliable monitoring posture in 2026 has six properties. First, every production LLM call is wrapped in a trace with at least one parent span and one model span. coverage above 99% for critical routes. Second, every span has a prompt.version tag so a regression can be attributed to the exact rollout that caused it. Third, evaluator scores live on the span itself, not in a separate dataset, so alerts can combine quality and runtime conditions. Fourth, dashboards slice by cohort. tenant, route, model, agent role. never by single global average. Fifth, the alert routes to the engineer who owns the failing component, not a shared firehose channel. Sixth, the gateway is in the data path so the team can route around a degraded provider while the post-mortem is still being written. Tools that hit four out of six are common; tools that hit all six are rare, and FutureAGI is built around all six.

Monitoring multi-agent systems is its own discipline

When a request fans out to a planner agent, a research agent, a synthesis agent, and a critic agent, the joint trace can have 40 spans and four sub-trajectories. Monitoring that stack requires per-agent slicing (agent.trajectory.step plus agent name), handoff-failure-rate tracking, per-role evaluators (ReasoningQuality for the planner, ToolSelectionAccuracy for the executor, Faithfulness for the synthesizer), and a joint TaskCompletion over the whole run. In our 2026 evals, teams that monitor only the joint TaskCompletion score miss the per-agent regressions that explain why team success rate moves. The cross-agent trace is the unit of debugging. see multi-agent system for the architecture and agent observability for the tracing pattern.

Drift detection in 2026: not just input-distribution shift

Drift in 2026 is multi-dimensional. Input drift (the prompts users send) still matters but is the easiest case. Output drift (the responses your model produces) is the new front-line signal because provider weight refreshes shift output style, length, and refusal rate without any client-side change. Evaluator drift. Groundedness rolling mean sliding 0.05 points week-over-week. predicts user complaints before they arrive. Retrieval drift, where a vector database reindex changes which chunks rank highest, breaks RAG quality silently. FutureAGI’s drift monitoring tracks all four classes by default and surfaces the suspect cohort, not just a global alarm. Compared to a classic model monitoring tool built for tabular ML, the LLM drift surface is wider, faster-moving, and requires evaluator-based detection rather than KS-test-style distribution checks.

Common mistakes (May 2026 edition)

• Watching only infrastructure metrics. CPU, HTTP status, and request latency do not explain prompt drift, retrieval errors, tool misuse, or hallucinated answers. AI-native monitoring is a different stack on top of APM, not a replacement.
• Aggregating away the trace. A weekly average hides the single route, tenant, prompt version, or tool that caused the incident. Always slice by cohort.
• Sampling without failure bias. Uniform 1% sampling misses rare expensive traces; sample failed evals, long spans, high-cost requests, and user complaints first, then add a uniform baseline.
• Treating monitoring as offline evals. Offline regression tests protect release gates; monitoring catches provider weight refreshes, RAG corpus drift, and live traffic shifts that offline tests cannot see.
• Alerting without ownership. Every threshold needs an owner, a review queue, and a clear next action. rollback, model fallback, guardrail tightening, or dataset repair. A threshold that fires into a Slack channel nobody owns is theater.
• Skipping prompt-version tags. Without prompt.version on every span, A/B rollouts and post-mortem regression attribution become guesswork.
• Trusting a single-signal alert. In our 2026 evals, joint-condition alerts cut false-positive rate by 60–80% relative to single-metric alerts.
• Ignoring refusal-rate drift. A model that quietly starts refusing 4% of valid questions after a weight refresh is a worse outage than a 500. and APM cannot see it.
• Skipping redaction. Prompts and completions carry PII, secrets, and customer data. Use PII or ProtectFlash pre-storage for any regulated workload.