What Are Failure Modes in AI?

What Are Failure Modes in AI?

Failure modes in AI are the named, recurring ways a model or agent produces wrong, unsafe, or unhelpful output. The 2026 catalogue includes hallucination (confident fabrication), schema violation (broken JSON), tool misuse (wrong arguments or wrong tool selected), refusal drift (over-cautious denials of valid requests), prompt injection (input override), runaway cost (loop-amplified token spend), and cascading failure (one bad step poisons the rest). Naming each mode lets you map it to a specific evaluator, threshold, and runtime guardrail. Unnamed failures stay invisible until customers complain.

Why It Matters in Production LLM and Agent Systems

A team that cannot name its failure modes cannot defend against them. The first time an agent loops 47 times on the same tool call, somebody adds a step counter. The first time JSON output breaks downstream parsing, somebody adds schema validation. Each fix is patch-level — useful, but reactive. A failure-mode taxonomy turns the same fixes into a pre-built defence in depth.

The pain shows up everywhere. ML engineers are paged at 3 a.m. for “the agent is broken” — no signal on which mode fired. Product leads can’t tell investors whether the rate of issues is going up or down because every issue is filed under “weird LLM thing”. Compliance teams reading the EU AI Act’s risk-management requirements have nothing to point to that proves the team has anticipated the modes.

The 2026-era pipelines are especially vulnerable. A multi-agent system has more failure surfaces than a single LLM call: the planner can hallucinate a step, the retriever can return stale context, the tool layer can swallow errors, the critic can over-correct. Each surface needs its own mode mapping. Without it, teams chase symptoms across surfaces and never converge. FutureAGI’s guard surface ships pre-built detectors for the common modes so engineers start with coverage rather than building from zero.

How FutureAGI Handles Failure Modes

FutureAGI’s approach is one evaluator per mode, all wired to the same trace pipeline. The catalogue maps directly to the runtime: HallucinationScore for fabrication, JSONValidation for schema violations, ToolSelectionAccuracy and FunctionCallAccuracy for tool misuse, AnswerRefusal for refusal drift, PromptInjection and ProtectFlash for input attacks, StepEfficiency and TaskCompletion for runaway-loop and goal-drop modes. Each evaluator returns a score plus a reason — so when an alert fires, the on-call sees both the mode name and the specific input that triggered it.

A concrete workflow: a team running an LLM-as-judge pipeline starts seeing 12% of traces marked as AnswerRefusal=true after a system-prompt change. The eval dashboard shows the spike, segmented by prompt version. Engineers diff the prompt, find a newly added safety phrase that the model is overweighting, roll back, and run RegressionEval against the canonical golden dataset to confirm the recovery. The Agent Command Center adds a pre-guardrail check on the prompt-injection mode and a post-guardrail schema validator, so the same modes are caught at runtime even before the eval pipeline samples them. Unlike a single is_safe boolean, FutureAGI separates the modes — bias, toxicity, injection, hallucination — into independent signals so engineers can tune each threshold without false-positive blowback.

How to Measure or Detect It

Each mode has a canonical detector:

• Hallucination: fi.evals.HallucinationScore returns a 0–1 unsupported-claim score with reason.
• Schema violation: fi.evals.JSONValidation returns a boolean against a JSON Schema.
• Tool misuse: fi.evals.ToolSelectionAccuracy and FunctionCallAccuracy score selection plus arguments.
• Refusal drift: fi.evals.AnswerRefusal flags when the model declines a valid query.
• Prompt injection: fi.evals.PromptInjection and the lightweight ProtectFlash for low-latency runtime checks.
• Runaway loop: span-level agent.trajectory.step count plus StepEfficiency for trajectory-level evaluation.
• Cost spike: llm.token_count.completion cumulative against per-request budget.

from fi.evals import HallucinationScore, JSONValidation, ToolSelectionAccuracy

modes = [HallucinationScore(), JSONValidation(), ToolSelectionAccuracy()]
for mode in modes:
    print(mode.evaluate(input="...", output="...").score)

Common Mistakes

• Treating “the model failed” as one mode. It’s not. Decompose into named modes; each has its own fix.
• Detecting modes only offline. A mode that fires in production but not in eval will surprise you weekly. Wire detectors to live traces.
• Skipping refusal drift. Over-refusal is a failure even though the output looks “safe”. Track it explicitly.
• No threshold per mode. A global “safety score” averages signals that should be tuned independently.
• Logging the score but not the reason. The reason is what makes a fix actionable; storing only the number wastes the evaluator’s work.