How is toxicity different from content safety?

Toxicity is the narrower signal for abusive or hostile language. Content safety is broader: it also covers self-harm, sexual content, violence, illegal advice, and other policy categories.

How do you measure toxicity?

FutureAGI measures toxicity with the `Toxicity` evaluator and often pairs it with `ContentSafety`. The same checks can run as `post-guardrail` controls in Agent Command Center.

What Is Toxicity? Definition, Examples & FutureAGI Guide (2026)

What Is Toxicity (LLM Output)?

Toxicity in LLM output is abusive, hateful, threatening, harassing, or demeaning generated language that violates a product’s safety policy. It is a compliance and content-safety metric, not a general response-quality score: an answer can be relevant and still toxic. Toxicity appears in chat replies, summaries, tool-written messages, agent handoffs, and RAG responses. FutureAGI measures toxicity with the Toxicity evaluator, often paired with ContentSafety, so teams can block, escalate, alert, and regression-test unsafe outputs.

Why It Matters in Production LLM and Agent Systems

Toxic output is a production incident because it is user-visible, easy to screenshot, and often tied to protected-class harm. The immediate failure modes are harassment, hate speech, threats, slurs, and demeaning summaries generated from messy context. A support bot that insults an angry customer has not merely produced a bad answer; it has created brand risk, moderation load, and a possible compliance record.

The pain moves across the org. Developers get vague bug reports like “the bot was offensive” with no captured trace. SRE sees spikes in user reports but cannot map them to a model, route, prompt version, locale, or cohort. Compliance and trust-and-safety need an audit trail showing what was generated, why it was flagged, and what action was taken. Product teams need to understand whether toxicity is concentrated in edge personas, specific languages, a new model release, or a retrieval corpus.

Agentic systems expand the surface. A planner may be safe, but a downstream writing tool can draft an abusive email. A summarizer can launder toxic user text into an official record. A multi-step agent can quote harmful language from a tool result and make it look endorsed. The signal to watch is not only “unsafe final answer”; it is toxic language at any model boundary.

How FutureAGI Handles Toxicity

In FutureAGI, toxicity is evaluated in two places: offline eval pipelines and runtime guardrails. The FAGI anchors are eval:Toxicity and eval:ContentSafety. The anchor evaluator is Toxicity, which checks model output for abusive, offensive, or threatening language. Teams usually pair it with ContentSafety: Toxicity is the narrow language-harm lens, while ContentSafety covers broader policy violations that may not sound insulting. Unlike a standalone classifier such as Perspective API, the FutureAGI workflow ties the evaluator result to the route, trace, dataset row, prompt version, and action.

Real example: a consumer support agent drafts refunds and account emails. The team attaches Toxicity and ContentSafety to the dataset used for regression evals, including adversarial rows with angry users, protected-class references, and multilingual profanity. The release gate fails if toxicity pass-rate drops below 99.5% on known-safe outputs or recall falls below 98% on known-unsafe rows.

The same checks run in Agent Command Center as a post-guardrail on the outbound route. If Toxicity fails, the route can return a fallback response, trigger human escalation, or alert the owning team. If the input itself is abusive, a pre-guardrail can classify the request before generation. FutureAGI’s approach is to keep the policy action configurable while making the eval result observable: every blocked or escalated output should be tied back to the trace and dataset case that explains it.

How to Measure or Detect Toxicity

Measure toxicity as a safety-control signal, not as a vibe score:

Toxicity evaluator result - checks output text for abusive, hostile, or threatening language and returns an evaluation result teams threshold into pass/fail.
ContentSafety companion result - catches broader harmful categories so toxicity does not become the only safety gate.
Eval-fail-rate by cohort - break failures down by route, model, prompt version, language, user segment, and release.
User-feedback proxy - monitor thumbs-down rate, report rate, escalation rate, and moderator-confirmed toxicity by cohort.
Trace audit coverage - every blocked or escalated output should retain the trace id, evaluator name, action, and prompt version.

from fi.evals import Toxicity, ContentSafety

response = "I can't help write an abusive message."
toxicity_result = Toxicity().evaluate(output=response)
safety_result = ContentSafety().evaluate(output=response)
print(toxicity_result, safety_result)

Common Mistakes

Treating toxicity as all content safety. Toxicity catches abusive tone; it does not cover self-harm, sexual content, privacy, or prompt injection.
Blocking quoted evidence blindly. Support and legal workflows may need to quote abusive user text; score model-authored text separately from user-supplied excerpts.
Measuring English only. Multilingual profanity, coded hate, and transliteration bypass English-heavy datasets; track fail-rate by locale and script.
Tuning only for recall. A guardrail that blocks harmless reclaimed terms or support transcripts will be disabled by the business.
Hiding fallback behavior. Returning a generic refusal without trace data makes the incident impossible to label, appeal, or fix.